[GRASS-dev] proposal for grass extensions and addons

Benjamin and William,

You both are trying to deal with the important concept of making easily available additional extensions to the standard GRASS distribution. If this can be made straightforward and consistent, we all hope that more people will consider creating many binaries or scripts that can extend GRASS’s capabilities in new ways. As you both have indicated, it is desirable to have any extension modules show up in the GUI menus to make them more easily accessible to users. Of course, any individual user may or may not have extensions and the extension she/he does may vary from those of others. So there needs to be a way of getting these into the GUI dynamically. Kind of borrowing from both of your initial ad hoc solutions, I’d like to propose the following way of dealing with extensions and getting them into the menus. I’m also trying to use the structure already existing in GRASS to avoid adding new complexities more than we need.

Anyone who wants to have extensions (binary modules or scripts) that show up in the menu, would need to…

  1. Put them in a directory and then make the directory visible to GRASS by adding it to GRASS_ADDON_PATH
  2. Create a simple text file named “xtnmenu.dat” the same directory as the extensions (you all can suggest another name if you like, but it has to be standardized).
  3. In xtnmenu.dat, list the executable names of each extension (i.e., the name you would type on the command line to start it) with each name separated from the others by a . All you need is the name. This way the TclTk GUI OR wxPython GUI can build their respective menu entries as they need them.
  4. A special name, called “separator”, would also be allowed to mark places where the list of names is divided by a horizontal line from other names in the menu.

At startup, the GUI would scan any directories stored in GRASS_ADDON_PATH searching for files named xtnmenu.dat. It would read each file and construct a menu entry from each item in the list of items in the file.

If all the extensions are in directories listed in the path statement of GRASS_ADDON_PATH, there is no need to worry about GRASS being able to find them to execute.

It would be easier to parse if everyone put the name of their extension(s) into the a SINGLE xtnmenu.dat file, shared by all (certainly my preference). But we should be able to deal with multiple xtnmenu.dat files, if need be, as long as they are all named xtnmenu.dat (or other standard) and all in a directory specified by GRASS_ADDON_PATH.

How does this sound?

Michael


Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

phone: 480-965-6213
fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton

Hi, just some comments inline.
2007/9/19, Michael Barton <michael.barton@asu.edu>:

1) Put them in a directory and then make the directory visible to GRASS by
adding it to GRASS_ADDON_PATH

IMHO there should be also one fixed add-on directory for system-wide
add-on's installed by root ($GISBASE/addons). Thus - root can install
add-on for all system users, user can install add-on only for self.

It would be easier to parse if everyone put the name of their extension(s)
into the a SINGLE xtnmenu.dat file, shared by all (certainly my preference).
But we should be able to deal with multiple xtnmenu.dat files, if need be,
as long as they are all named xtnmenu.dat (or other standard) and all in a
directory specified by GRASS_ADDON_PATH.

Single menu.dat (or whatever):
- single menu.dat (xtnmenu.dat) file requires user to use some menu
editor or digg into cryptic configuration file manualy to add new
add-on (if not some GEM-like feature is used).
+ User can have custom ordered menu instead of auto-generated one.

I'm not following closely to recent movement in GRASS_ADDONS and thus
have no idea how add-on directory looks like (wiki also has no entry
for it), but there could be two approaches (just thinking):
1) single directory for all add-on's.
  + no mess with different directories etc.
  - requires additional checks before adding new add-on does file
names not collide (like i.e. help.html)
2) directory per add-on. ($GRASS_ADDON_PATH/add-on_name/bin|etc|man|)
  + binary-compatible add-on's can be added just by untaring in to
$GRASS_ADDON_PATH;
  - mess with multiple directories;

In any way - if single menu conf file approach is choosed, then menu
editor is a must.

Did I missed how add-on icon is defined? If it's not used now, it
could be used in future.

How does this sound?

Any attempt to standartize unclear GRASS areas is good.

Michael

Sorry for interruption and my English,
Maris.

On Sep 18, 2007, at 11:31 PM, Michael Barton wrote:

Anyone who wants to have extensions (binary modules or scripts) that show up in the menu, would need to...

1) Put them in a directory and then make the directory visible to GRASS by adding it to GRASS_ADDON_PATH

Currently available. It's the path(s) to extra bin folders.

3) In xtnmenu.dat, list the executable names of each extension (i.e., the name you would type on the command line to start it) with each name separated from the others by a <newline>. All you need is the name. This way the TclTk GUI OR wxPython GUI can build their respective menu entries as they need them.

How about 3 items per command - the menu name, help text (can be empty) and command.

4) A special name, called "separator", would also be allowed to mark places where the list of names is divided by a horizontal line from other names in the menu.

Submenus, if only one level, would be a good idea. If a user has a lot of extras installed, or just puts all their scripts in the menu, the menu could get very long.

At startup, the GUI would scan any directories stored in GRASS_ADDON_PATH searching for files named xtnmenu.dat. It would read each file and construct a menu entry from each item in the list of items in the file.

The menu files should probably NOT be in the same dir as the binaries themselves. That's what I set up the GRASS_ADDON_ETC for - it's where any programs that need extra data files look for them, including GRASS itself.

It would be easier to parse if everyone put the name of their extension(s) into the a SINGLE xtnmenu.dat file, shared by all (certainly my preference). But we should be able to deal with multiple xtnmenu.dat files, if need be, as long as they are all named xtnmenu.dat (or other standard) and all in a directory specified by GRASS_ADDON_PATH.

How about two default files:

- xtnmenu.dat for auto-generated menu items, either from GEM installer or something like my current startup method

- xtnmenu_user.dat for the user to maunally edit, without being trashed by the auto-generating methods.

When you think about it, ALL extensions (ie those installed by GEM and those by other methods) don't need to be in the same file (that could be trouble maintaining) - they just each use a different path in GRASS_ADDON_ETC.

-----
William Kyngesburye <kyngchaos*at*kyngchaos*dot*com>
http://www.kyngchaos.com/

[Trillian] What are you supposed to do WITH a maniacally depressed robot?

[Marvin] You think you have problems? What are you supposed to do if you ARE a maniacally depressed robot? No, don't try and answer, I'm 50,000 times more intelligent than you and even I don't know the answer...

- HitchHiker's Guide to the Galaxy

On Sep 19, 2007, at 2:28 AM, Maris Nartiss wrote:

Hi, just some comments inline.
2007/9/19, Michael Barton <michael.barton@asu.edu>:

1) Put them in a directory and then make the directory visible to GRASS by
adding it to GRASS_ADDON_PATH

IMHO there should be also one fixed add-on directory for system-wide
add-on's installed by root ($GISBASE/addons). Thus - root can install
add-on for all system users, user can install add-on only for self.

This should probably be left to a user/packager option - let them decide what it should be. ie on OSX installing any extras directly in an app bundle is wrong. Thus I currently have the GRASS startup for OSX (app build, not unix build) add an external global path to GRASS_ADDON_PATH.

GEM already takes care of installing inside the GRASS distribution.

It would be easier to parse if everyone put the name of their extension(s)
into the a SINGLE xtnmenu.dat file, shared by all (certainly my preference).
But we should be able to deal with multiple xtnmenu.dat files, if need be,
as long as they are all named xtnmenu.dat (or other standard) and all in a
directory specified by GRASS_ADDON_PATH.

Single menu.dat (or whatever):
- single menu.dat (xtnmenu.dat) file requires user to use some menu
editor or digg into cryptic configuration file manualy to add new
add-on (if not some GEM-like feature is used).
+ User can have custom ordered menu instead of auto-generated one.

I'm not following closely to recent movement in GRASS_ADDONS and thus
have no idea how add-on directory looks like (wiki also has no entry
for it), but there could be two approaches (just thinking):

So far it's something I've worked out in the OSX app builds, but could easily be extended to other platforms (I just haven't been aggressive in pushing it to other platforms).

In any way - if single menu conf file approach is choosed, then menu
editor is a must.

That's a nice idea - an extension menu editor in the GUI. Probably should only be used for the user menu file (as I proposed having replying to Michael) and not for auto-generated menus. Then the user wouldn't have to dig for the menu file.

-----
William Kyngesburye <kyngchaos*at*kyngchaos*dot*com>
http://www.kyngchaos.com/

"I ache, therefore I am. Or in my case - I am, therefore I ache."

- Marvin

On 9/19/07 7:12 AM, "William Kyngesburye" <woklist@kyngchaos.com> wrote:

On Sep 19, 2007, at 2:28 AM, Maris Nartiss wrote:

Hi, just some comments inline.
2007/9/19, Michael Barton <michael.barton@asu.edu>:

1) Put them in a directory and then make the directory visible to
GRASS by
adding it to GRASS_ADDON_PATH

IMHO there should be also one fixed add-on directory for system-wide
add-on's installed by root ($GISBASE/addons). Thus - root can install
add-on for all system users, user can install add-on only for self.

This should probably be left to a user/packager option - let them
decide what it should be. ie on OSX installing any extras directly
in an app bundle is wrong. Thus I currently have the GRASS startup
for OSX (app build, not unix build) add an external global path to
GRASS_ADDON_PATH.

Whatever directories used for extensions (fewer is better than more), it
should NOT be $GISBASE/[something].

Anything in $GISBASE will be overwritten with a binary install on Mac and
Windows (I'm not sure on Linux). It is better to have extensions live
outside of the main GRASS directory structure so that they persist through
updates.

It's OK with me to require a single directory name for all extensions, but
allow it to live in different places on different systems.

Probably the most flexible would be to allow extension modules to live
anywhere, as long as they are in GRASS_ADDON_PATH. A single menu def file
would live in GRASS_ADDON_EXT.

GEM already takes care of installing inside the GRASS distribution.

This is a problem for binary installations.

It would be easier to parse if everyone put the name of their
extension(s)
into the a SINGLE xtnmenu.dat file, shared by all (certainly my
preference).
But we should be able to deal with multiple xtnmenu.dat files, if
need be,
as long as they are all named xtnmenu.dat (or other standard) and
all in a
directory specified by GRASS_ADDON_PATH.

Single menu.dat (or whatever):
- single menu.dat (xtnmenu.dat) file requires user to use some menu
editor or digg into cryptic configuration file manualy to add new
add-on (if not some GEM-like feature is used).

No need to have cryptic configuration files. I like William's suggestion.
menu entry, command name, help text. Could be separated by commas or colons.
No need for anything else.

While the possibility of multiple menu def files avoids conflicts between
command names within a file, it doesn't help in overall function when you
actually try to run something. A single menu files would force people to
deal with it.

+ User can have custom ordered menu instead of auto-generated one.

I'm not following closely to recent movement in GRASS_ADDONS and thus
have no idea how add-on directory looks like (wiki also has no entry
for it), but there could be two approaches (just thinking):

So far it's something I've worked out in the OSX app builds, but
could easily be extended to other platforms (I just haven't been
aggressive in pushing it to other platforms).

In any way - if single menu conf file approach is choosed, then menu
editor is a must.

Not a *must* if the file is simple, but could be handy. Before a menu editor
(or even menu cascades) get worked out, we simply need to agree on a
protocol that can be implemented consistently.

That's a nice idea - an extension menu editor in the GUI. Probably
should only be used for the user menu file (as I proposed having
replying to Michael) and not for auto-generated menus. Then the user
wouldn't have to dig for the menu file.

After some reflection, I guess I'd rather see a single extension menu def
file that could be customized by users. The main reason is to avoid
conflicts. Anyone who has a system to automatically add entries to such a
file would have the responsibility to check for possible conflicts in menu
item and executable name. This would be easier in a single file than if
there could be multiple files.

Michael

-----
William Kyngesburye <kyngchaos*at*kyngchaos*dot*com>
http://www.kyngchaos.com/

"I ache, therefore I am. Or in my case - I am, therefore I ache."

- Marvin

__________________________________________
Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

phone: 480-965-6213
fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton

Whatever we do, I'd strongly argue for an approach that keeps
all differences between operating systems hidden from the user.

We should also aim for one unified mechanism and extension format,
as we will not be able to keep more than one supported in the long term.

Also, keeping complex, potentially cascaded menu entries in an
ASCII file makes it hard to parse that file, add (and delete,
update!) entries as necessary without corrupting the file structure,
duplicating or removing things accidentally etc. It involves creating
a customized parser from scratch.

The only approach I can think of to keep it file-based would be
to use an XML file database. We can use the GPL's libXML2 for that,
but this would mean introducing one more dependency to GRASS ;(

As an alternative, consider using folders and files to create the
menu structure:

- all the user needs to do is point GRASS to a top-level folder
for menu entries. Default, system-wide and user plug-ins folder
can easily be provided.

- the GUI than goes to that "master" folder and looks for subfolders.
Every folder inside the master is a new MENU entry (can be
n subfolders deep to allow cascading menus). Ascii files
inside a subfolder can represent simple menu ITEMS (one per
file). These will be added to the current menu level alphabetically
by file name (but item name as shown in the menu could be anything as
defined in the ascii file).

- adding entries for a new extension amounts to simply creating folders
inside the master folder. Removing them is just removing the folder(s).
Updating is just overwriting and existing folder (possibly first deleting the old one), etc. Very simple to do. Can all be handled
transparently using system's file commands.

- all the GUI needs to do is go thru the folders and subfolders and
piece together the menu cascades. Also very easy.

Benjamin

Michael Barton wrote:

Benjamin and William,

You both are trying to deal with the important concept of making easily available additional extensions to the standard GRASS distribution. If this can be made straightforward and consistent, we all hope that more people will consider creating many binaries or scripts that can extend GRASS's capabilities in new ways. As you both have indicated, it is desirable to have any extension modules show up in the GUI menus to make them more easily accessible to users. Of course, any individual user may or may not have extensions and the extension she/he does may vary from those of others. So there needs to be a way of getting these into the GUI dynamically. Kind of borrowing from both of your initial ad hoc solutions, I'd like to propose the following way of dealing with extensions and getting them into the menus. I'm also trying to use the structure already existing in GRASS to avoid adding new complexities more than we need.

Anyone who wants to have extensions (binary modules or scripts) that show up in the menu, would need to...

1) Put them in a directory and then make the directory visible to GRASS by adding it to GRASS_ADDON_PATH
2) Create a simple text file named "xtnmenu.dat" the same directory as the extensions (you all can suggest another name if you like, but it has to be standardized).
3) In xtnmenu.dat, list the executable names of each extension (i.e., the name you would type on the command line to start it) with each name separated from the others by a <newline>. All you need is the name. This way the TclTk GUI OR wxPython GUI can build their respective menu entries as they need them.
4) A special name, called "separator", would also be allowed to mark places where the list of names is divided by a horizontal line from other names in the menu.

At startup, the GUI would scan any directories stored in GRASS_ADDON_PATH searching for files named xtnmenu.dat. It would read each file and construct a menu entry from each item in the list of items in the file.

If all the extensions are in directories listed in the path statement of GRASS_ADDON_PATH, there is no need to worry about GRASS being able to find them to execute.

It would be easier to parse if everyone put the name of their extension(s) into the a SINGLE xtnmenu.dat file, shared by all (certainly my preference). But we should be able to deal with multiple xtnmenu.dat files, if need be, as long as they are all named xtnmenu.dat (or other standard) and all in a directory specified by GRASS_ADDON_PATH.

How does this sound?

Michael
__________________________________________
Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change Center for Social Dynamics & Complexity
Arizona State University

phone: 480-965-6213
fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton

Benjamin,

These all seem like good ideas as a directions to head.

I'd like to take this a step at a time, however, making sure that we keep
open the options to make the next step each time.

On 9/19/07 1:20 PM, "Benjamin Ducke" <benjamin.ducke@ufg.uni-kiel.de> wrote:

Whatever we do, I'd strongly argue for an approach that keeps
all differences between operating systems hidden from the user.

Definitely agree

We should also aim for one unified mechanism and extension format,
as we will not be able to keep more than one supported in the long term.

Also agree sort of. There are different ways of *making* extensions,
scripts, the GEM approach, William's makefile approach, and certainly
others. It's probably a good idea to settle on a limited number of these
(e.g., decide on the best way to compile a binary from source, and a best
way to distribute a binary, etc.).

But at the moment, I think it would be good to first focus on what we do
with extensions however we make them. For this, we need some kind of
standard that makes them easy to use from the command line and easy to run
as menu items in the GUI.

If we get this far, we can use the experience to go the next steps.

Also, keeping complex, potentially cascaded menu entries in an
ASCII file makes it hard to parse that file, add (and delete,
update!) entries as necessary without corrupting the file structure,
duplicating or removing things accidentally etc. It involves creating
a customized parser from scratch.

I agree here too. To begin with, let's just start with a protocol that will
generate a flat menu. We can try not to close off options to make this more
complex in the future. However, at the moment, there are not such a huge
number of extensions that this will be a problem in most cases.

The only approach I can think of to keep it file-based would be
to use an XML file database. We can use the GPL's libXML2 for that,
but this would mean introducing one more dependency to GRASS ;(

With the switch to wxPython, XML will become a viable option without any new
dependencies. Until then, it would be good not to introduce new dependencies
to the extent possible.

As an alternative, consider using folders and files to create the
menu structure:

The folder structure is elegant for creating cascading menus. However, it
has some complications for actually holding extensions that can be
run--especially from the command line. If a directory holding an extension
is listed in GRASS_ADDON_PATH, it can be run by simply typing its name from
the command line. If you put extensions into a nested hierarchy of folders,
they would all have to go into GRASS_ADDON_PATH to run the extension from
the command line. This could quickly get unwieldy.

In addition to the automated way of creating menus when extensions are
installed by an instalation package like GEM, it seems a good idea to also
be able to support the user who makes a couple of special scripts and wants
to put them in an extensions directory and get them listed on the menu.

Michael

- all the user needs to do is point GRASS to a top-level folder
for menu entries. Default, system-wide and user plug-ins folder
can easily be provided.

- the GUI than goes to that "master" folder and looks for subfolders.
Every folder inside the master is a new MENU entry (can be
n subfolders deep to allow cascading menus). Ascii files
inside a subfolder can represent simple menu ITEMS (one per
file). These will be added to the current menu level alphabetically
by file name (but item name as shown in the menu could be anything as
defined in the ascii file).

- adding entries for a new extension amounts to simply creating folders
inside the master folder. Removing them is just removing the folder(s).
Updating is just overwriting and existing folder (possibly first
deleting the old one), etc. Very simple to do. Can all be handled
transparently using system's file commands.

- all the GUI needs to do is go thru the folders and subfolders and
piece together the menu cascades. Also very easy.

Benjamin

Michael Barton wrote:

Benjamin and William,

You both are trying to deal with the important concept of making easily
available additional extensions to the standard GRASS distribution. If
this can be made straightforward and consistent, we all hope that more
people will consider creating many binaries or scripts that can extend
GRASS's capabilities in new ways. As you both have indicated, it is
desirable to have any extension modules show up in the GUI menus to make
them more easily accessible to users. Of course, any individual user
may or may not have extensions and the extension she/he does may vary
from those of others. So there needs to be a way of getting these into
the GUI dynamically. Kind of borrowing from both of your initial ad hoc
solutions, I'd like to propose the following way of dealing with
extensions and getting them into the menus. I'm also trying to use the
structure already existing in GRASS to avoid adding new complexities
more than we need.

Anyone who wants to have extensions (binary modules or scripts) that
show up in the menu, would need to...

1) Put them in a directory and then make the directory visible to GRASS
by adding it to GRASS_ADDON_PATH
2) Create a simple text file named "xtnmenu.dat" the same directory as
the extensions (you all can suggest another name if you like, but it has
to be standardized).
3) In xtnmenu.dat, list the executable names of each extension (i.e.,
the name you would type on the command line to start it) with each name
separated from the others by a <newline>. All you need is the name. This
way the TclTk GUI OR wxPython GUI can build their respective menu
entries as they need them.
4) A special name, called "separator", would also be allowed to mark
places where the list of names is divided by a horizontal line from
other names in the menu.

At startup, the GUI would scan any directories stored in
GRASS_ADDON_PATH searching for files named xtnmenu.dat. It would read
each file and construct a menu entry from each item in the list of items
in the file.

If all the extensions are in directories listed in the path statement of
GRASS_ADDON_PATH, there is no need to worry about GRASS being able to
find them to execute.

It would be easier to parse if everyone put the name of their
extension(s) into the a SINGLE xtnmenu.dat file, shared by all
(certainly my preference). But we should be able to deal with multiple
xtnmenu.dat files, if need be, as long as they are all named xtnmenu.dat
(or other standard) and all in a directory specified by GRASS_ADDON_PATH.

How does this sound?

Michael
__________________________________________
Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

phone: 480-965-6213
fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton

__________________________________________
Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

phone: 480-965-6213
fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton

On Sep 19, 2007, at 9:36 PM, Michael Barton wrote:

We should also aim for one unified mechanism and extension format,
as we will not be able to keep more than one supported in the long term.

Also agree sort of. There are different ways of *making* extensions,
scripts, the GEM approach, William's makefile approach, and certainly
others. It's probably a good idea to settle on a limited number of these
(e.g., decide on the best way to compile a binary from source, and a best
way to distribute a binary, etc.).

But at the moment, I think it would be good to first focus on what we do
with extensions however we make them. For this, we need some kind of
standard that makes them easy to use from the command line and easy to run
as menu items in the GUI.

If we get this far, we can use the experience to go the next steps.

Glynn mentioned in the v.in.dwg discussion that the makefile fragments present in a 6.3 install are meant to enable building modules from source without the full GRASS source. I had noticed these some time ago. I need to try it out - it's basically what I'm doing with my external build makefiles.

-----
William Kyngesburye <kyngchaos*at*kyngchaos*dot*com>
http://www.kyngchaos.com/

"Oh, look, I seem to have fallen down a deep, dark hole. Now what does that remind me of? Ah, yes - life."

- Marvin

William Kyngesburye wrote:

Glynn mentioned in the v.in.dwg discussion that the makefile
fragments present in a 6.3 install are meant to enable building
modules from source without the full GRASS source.

At present, using it is probably a bit more complex than it needs to
be. The *.make files still contain some references to the source
directory, which need to be overridden. Specifically, Platform.make
has:

  GRASS_HOME = /usr/local/src/grass/cvs
  RUN_GISBASE = /usr/local/src/grass/cvs/dist.i686-pc-linux-gnu

RUN_GISBASE is used for the dummy session for running GRASS commands
during the build process, for generating the HTML files (Html.make)
and getting strings to translate from scripts (Script.make). In both
cases, it's used like:

  GISRC=$(RUN_GISBASE)/demolocation/.grassrc${GRASS_VERSION_MAJOR}${GRASS_VERSION_MINOR} \
  GISBASE=$(RUN_GISBASE) \

Note that the demolocation directory *isn't* installed.

GRASS_HOME is used in a number of places; Grass.make has:

  ARCH_DISTDIR = $(GRASS_HOME)/dist.$(ARCH)
  ARCH_BINDIR = $(GRASS_HOME)/bin.$(ARCH)
  GRASS_INCDIR = $(GRASS_HOME)/include
  GRASS_LIBDIR = $(GRASS_HOME)/lib

  ...

  ifdef INST_NOW
  ARCH_DISTDIR = $(INST_DIR)
  ARCH_BINDIR = $(UNIX_BIN)
  endif

While Dir.make has:

  $(MAKE) -C $$subdir || echo $(CURDIR)/$$subdir >> $(GRASS_HOME)/error.log; \

and Script.make has:

  $(BIN)/$(PGM).bat: $(GRASS_HOME)/scripts/windows_launch.bat
    sed -e "s#SCRIPT_NAME#$(PGM)#" $(GRASS_HOME)/scripts/windows_launch.bat > $@

If you forget about Script.make and Html.make for now, you will
probably need to use something like:

  make ... GRASS_HOME=. INST_NOW=y MODULE_TOPDIR=/usr/local/grass-6.3.cvs

The other variables are less important.

ARCH_BINDIR is only used for the grass63[.bat] script.

GRASS_INCDIR is erroneously used for dependencies in a small number of
Makefiles (the actual -I switches refer to ARCH_INCDIR, which is
dist.<arch>/include).

GRASS_LIBDIR is only used to find a couple of DBMI headers using:

  -I$(GRASS_LIBDIR)/db/dbmi_driver

As for fixes:

GRASS_INCDIR and GRASS_LIBDIR can probably be eliminated relatively
easily.

RUN_GISBASE should be split into RUN_GISBASE and RUN_GISRC.

The demolocation directory should be installed, along with a grassrc
file which refers to it.

There should be a specific ERRORLOG variable for error.log.

INST_NOW=y should set RUN_GISBASE and RUN_GISRC to point to the
installed versions, and ERRORLOG to "./error.log".

--
Glynn Clements <glynn@gclements.plus.com>

This looks familiar from when I first noticed the makefile bits getting installed. I work around most of it in my OSX modbuild setup by modifying grass.make and platform.make.

It would be nice to get it cleaned up and working - no need for my OSX hack.

A big problem I see is that INST_NOW. That's just what I need to avoid for the OSX app build - installing (or building) into the binary distribution. I avoid that in my modbuild by having a local/external copy of the makefile bits in a folder layout like the source, so modules get built into the local distdir.

On Sep 20, 2007, at 4:36 AM, Glynn Clements wrote:

At present, using it is probably a bit more complex than it needs to
be. The *.make files still contain some references to the source
directory, which need to be overridden. Specifically, Platform.make
has:

  GRASS_HOME = /usr/local/src/grass/cvs
  RUN_GISBASE = /usr/local/src/grass/cvs/dist.i686-pc-linux-gnu

RUN_GISBASE is used for the dummy session for running GRASS commands
during the build process, for generating the HTML files (Html.make)
and getting strings to translate from scripts (Script.make). In both
cases, it's used like:

  GISRC=$(RUN_GISBASE)/demolocation/.grassrc${GRASS_VERSION_MAJOR}${GRASS_VERSION_MINOR} \
  GISBASE=$(RUN_GISBASE) \

Note that the demolocation directory *isn't* installed.

GRASS_HOME is used in a number of places; Grass.make has:

  ARCH_DISTDIR = $(GRASS_HOME)/dist.$(ARCH)
  ARCH_BINDIR = $(GRASS_HOME)/bin.$(ARCH)
  GRASS_INCDIR = $(GRASS_HOME)/include
  GRASS_LIBDIR = $(GRASS_HOME)/lib

  ...

  ifdef INST_NOW
  ARCH_DISTDIR = $(INST_DIR)
  ARCH_BINDIR = $(UNIX_BIN)
  endif

While Dir.make has:

  $(MAKE) -C $$subdir || echo $(CURDIR)/$$subdir >> $(GRASS_HOME)/error.log; \

and Script.make has:

  $(BIN)/$(PGM).bat: $(GRASS_HOME)/scripts/windows_launch.bat
    sed -e "s#SCRIPT_NAME#$(PGM)#" $(GRASS_HOME)/scripts/windows_launch.bat > $@

If you forget about Script.make and Html.make for now, you will
probably need to use something like:

  make ... GRASS_HOME=. INST_NOW=y MODULE_TOPDIR=/usr/local/grass-6.3.cvs

The other variables are less important.

ARCH_BINDIR is only used for the grass63[.bat] script.

GRASS_INCDIR is erroneously used for dependencies in a small number of
Makefiles (the actual -I switches refer to ARCH_INCDIR, which is
dist.<arch>/include).

GRASS_LIBDIR is only used to find a couple of DBMI headers using:

  -I$(GRASS_LIBDIR)/db/dbmi_driver

As for fixes:

GRASS_INCDIR and GRASS_LIBDIR can probably be eliminated relatively
easily.

RUN_GISBASE should be split into RUN_GISBASE and RUN_GISRC.

The demolocation directory should be installed, along with a grassrc
file which refers to it.

There should be a specific ERRORLOG variable for error.log.

INST_NOW=y should set RUN_GISBASE and RUN_GISRC to point to the
installed versions, and ERRORLOG to "./error.log".

--
Glynn Clements <glynn@gclements.plus.com>

-----
William Kyngesburye <kyngchaos*at*kyngchaos*dot*com>
http://www.kyngchaos.com/

First Pogril: Why is life like sticking your head in a bucket filled with hyena offal?
Second Pogril: I don't know. Why IS life like sticking your head in a bucket filled with hyena offal?
First Pogril: I don't know either. Wretched, isn't it?

-HitchHiker's Guide to the Galaxy

scripts, the GEM approach, William's makefile approach, and certainly
others. It's probably a good idea to settle on a limited number of these
(e.g., decide on the best way to compile a binary from source, and a best
way to distribute a binary, etc.).

GEM and its whole approach to packaging source code is actually a pretty
dirty hack. If William can come up with a cleaner, less fatty solution
of compiling modules outside the source tree -- I'll be happy to supplant that for what's currently in GEM.
Both William's approach and mine are essentially the same idea. It would
be not much work to make GEM use William's, potentially revised make
system instead of mine. We can also add support for customizable extension folder to GEM. I think GEM does a decent job as a GUI to the
installation process. However, we are still left with a few crucial issues:

1. Acquiring super-user permissions for installations (maybe not a problem any longer once we have user-definable extension dirs support)

2. Integrating extension modules' HTML man pages into main index without
using the GRASS Make system

3. Binary installs

With the switch to wxPython, XML will become a viable option without any new
dependencies. Until then, it would be good not to introduce new dependencies
to the extent possible.

Also, module installation should be possible, even if GRASS is installed w/o Python support!

The folder structure is elegant for creating cascading menus. However, it
has some complications for actually holding extensions that can be
run--especially from the command line. If a directory holding an extension
is listed in GRASS_ADDON_PATH, it can be run by simply typing its name from
the command line. If you put extensions into a nested hierarchy of folders,
they would all have to go into GRASS_ADDON_PATH to run the extension from
the command line. This could quickly get unwieldy.

I was actually thinking about ONLY putting the menu description files into these folders. Executables would stay in the respective bin directories, where they belong (be it $GRASS_ADDON_PATH or some global
system dir).

Benjamin

On Sep 20, 2007, at 3:04 PM, Benjamin Ducke wrote:

scripts, the GEM approach, William's makefile approach, and certainly
others. It's probably a good idea to settle on a limited number of these
(e.g., decide on the best way to compile a binary from source, and a best
way to distribute a binary, etc.).

GEM and its whole approach to packaging source code is actually a pretty
dirty hack. If William can come up with a cleaner, less fatty solution
of compiling modules outside the source tree -- I'll be happy to supplant that for what's currently in GEM.

See also Glynn's comment on the makefile frags already in the install. With what I've figured out for my method, Glynn's helpful comments, and your GEM work, I think we can work out something simple and flexible.

Both William's approach and mine are essentially the same idea. It would
be not much work to make GEM use William's, potentially revised make
system instead of mine. We can also add support for customizable extension folder to GEM. I think GEM does a decent job as a GUI to the
installation process. However, we are still left with a few crucial issues:

1. Acquiring super-user permissions for installations (maybe not a problem any longer once we have user-definable extension dirs support)

Indeed, part of the idea is for non-admin users to be able to build addons and install them in their home folder.

2. Integrating extension modules' HTML man pages into main index without
using the GRASS Make system

Right now, I use a separate help index from the installed/built one. I don't think there is a way with basic HTML files to dynamically include other html files. It could work if there was a local webserver, but that's not something we can count on.

For now, I make use of the Mac OS X help system to easily access the help files (though that isn't necessary in this context). Similar to building the GUI menu off the addons menu file(s), we could also build the help menu from additional help index files. Something would have to be done with the command line help also - it would have to check paths for module help files, something like $GRASS_ADDON_HELP. (Michael - I forgot that the help menu was another bit to work on along with the addon menu.)

Though with all these GRASS_ADDON_* variables, I wonder if we should simplify them into just GRASS_ADDON_PATH which would be the base for structured dir with bin, lib, docs, etc subdirs.

3. Binary installs

I was thinking about how to handle that. Even installs from a source build for a module. I didn't come up with anything, but it should be possible. For a make install for a module, pass the INST_DIR and let it override the configured INST_DIR. The same could be used for a binary distribution for a module.

-----
William Kyngesburye <kyngchaos*at*kyngchaos*dot*com>
http://www.kyngchaos.com/

First Pogril: Why is life like sticking your head in a bucket filled with hyena offal?
Second Pogril: I don't know. Why IS life like sticking your head in a bucket filled with hyena offal?
First Pogril: I don't know either. Wretched, isn't it?

-HitchHiker's Guide to the Galaxy

See also Glynn's comment on the makefile frags already in the install. With what I've figured out for my method, Glynn's helpful comments, and your GEM work, I think we can work out something simple and flexible.

This should not even require changing anything about GEM for the time
being. Just replace the makefiles I currently use for extensions
packages (see 'skeleton' dir that gets installed into 'etc' for GRASS
CVS installs) with the revised versions. GEM really just calls make
install and the rest is done by the make files included in the extension
package. I have documented this approach (and all make file changes I
made) in the GEM documentation accessible from the GRASS CVS help index.

2. Integrating extension modules' HTML man pages into main index without
using the GRASS Make system

Right now, I use a separate help index from the installed/built one. I don't think there is a way with basic HTML files to dynamically include other html files. It could work if there was a local webserver, but that's not something we can count on.

Right, GEM currently used a lot of HTML file parsing to manipulate the
main index.html (it looks for a special commented in there in order
to find the place to append its own index). However, for user installs,
we need a local HTML index, which needs to be linked to the system-wide
HTML index (however, in order to manipulate that one, we again would
need superuser rights -- aargh!).

It might be easier to create a second HTML index especially for
extensions which we can layout so that it becomes easier to parse for
our purposes. We could create a second GRASS script like g.manual.user
to display that index and put a not about it into the global HTML index.

Though with all these GRASS_ADDON_* variables, I wonder if we should simplify them into just GRASS_ADDON_PATH which would be the base for structured dir with bin, lib, docs, etc subdirs.

Yes. I would be in favour of that: one plugin dir for each user and
nothing more (to worry about).

3. Binary installs

I was thinking about how to handle that. Even installs from a source build for a module. I didn't come up with anything, but it should be possible. For a make install for a module, pass the INST_DIR and let it override the configured INST_DIR. The same could be used for a binary distribution for a module.

Yes. Only problem is that for a simple "make install" (which is what
GEM currently does for binaries!) the user needs to
have GNU make tools installed on the system. So it would be better to
code all the stuff we need in C and include that in GEM.
If we manage to sort out the issues with the plugin dir and the HTML
page merging, this should not be very hard to do.

Also, extensions need to ship with different binaries for different
systems. We could try and guess the right ones automatically for the
user but I don't suppose there is a portable do of doing that.

Benjamin

-----
William Kyngesburye <kyngchaos*at*kyngchaos*dot*com>
http://www.kyngchaos.com/

First Pogril: Why is life like sticking your head in a bucket filled with hyena offal?
Second Pogril: I don't know. Why IS life like sticking your head in a bucket filled with hyena offal?
First Pogril: I don't know either. Wretched, isn't it?

-HitchHiker's Guide to the Galaxy

On Sep 20, 2007, at 5:05 PM, Benjamin Ducke wrote:

2. Integrating extension modules' HTML man pages into main index without
using the GRASS Make system

Right now, I use a separate help index from the installed/built one. I don't think there is a way with basic HTML files to dynamically include other html files. It could work if there was a local webserver, but that's not something we can count on.

Right, GEM currently used a lot of HTML file parsing to manipulate the
main index.html (it looks for a special commented in there in order
to find the place to append its own index). However, for user installs,
we need a local HTML index, which needs to be linked to the system-wide
HTML index (however, in order to manipulate that one, we again would
need superuser rights -- aargh!).

The problem with having ANY link in the built/installed help index to a user index is that the URL for every user to their home folder is different. You can't use $HOME/ or ~/ in URLs (though I found that it does work in Safari, but that's an exception).

It might be easier to create a second HTML index especially for
extensions which we can layout so that it becomes easier to parse for
our purposes. We could create a second GRASS script like g.manual.user
to display that index and put a not about it into the global HTML index.

My OSX index generator does this (the creation part). See macosx/app/build_html_user_index.sh. But we shouldn't need a separate g.manual.user, g.manual should figure it out. For individual pages, it would look in dirs specified in GRASS_ADDON_HELP (or some such path variable), then the GRASS binary package. The first path it finds it in would be passed as the URL to the help browser.

For the index, that's a bit tricky from g.manual. Each path in the hypothetical GRASS_ADDON_HELP could have an index. So asking it to show the "user" index could be ambiguous. At least in the GUI, as many menu items can be used as are needed.

Though with all these GRASS_ADDON_* variables, I wonder if we should simplify them into just GRASS_ADDON_PATH which would be the base for structured dir with bin, lib, docs, etc subdirs.

Yes. I would be in favour of that: one plugin dir for each user and
nothing more (to worry about).

Restricting the number of addon dirs may simplify the whole thing (1 global, 1 user per user). But that's not quite the same flexibility that the rest of GRASS has. ie multiple paths are possible in GRASS_ADDON_PATH (as it is used currently), and the user is free to add to it. Glynn and others could probably expand on this.

-----
William Kyngesburye <kyngchaos*at*kyngchaos*dot*com>
http://www.kyngchaos.com/

All generalizations are dangerous, even this one.

William and Benjamin,

This is fantastic progress on working out a straightforward, unified
protocol to let users enhance their GRASS installations. I only have a few
comments. Below.

On 9/20/07 3:05 PM, "Benjamin Ducke" <benjamin.ducke@ufg.uni-kiel.de> wrote:

2. Integrating extension modules' HTML man pages into main index without
using the GRASS Make system

...

It might be easier to create a second HTML index especially for
extensions which we can layout so that it becomes easier to parse for
our purposes. We could create a second GRASS script like g.manual.user
to display that index and put a not about it into the global HTML index.

Though with all these GRASS_ADDON_* variables, I wonder if we should
simplify them into just GRASS_ADDON_PATH which would be the base for
structured dir with bin, lib, docs, etc subdirs.

Yes. I would be in favour of that: one plugin dir for each user and
nothing more (to worry about).

I don't know how g.manual currently searches for the main GRASS doc
directory, but it seems like it wouldn't be hard to change it to look in a
doc directory within GRASS_ADDON_PATH.

Like other extension files, this would survive a binary update on Mac and
Windows.

3. Binary installs

...

Also, extensions need to ship with different binaries for different
systems. We could try and guess the right ones automatically for the
user but I don't suppose there is a portable do of doing that.

Binaries would be highly desirable for many users when possible. But whether
or not they would get built should be up to the developer to hassle with. It
isn't realistic to try to create a binary for all possible systems, but a
limited number could cover a lot (e.g., RPM's, Debian, Mac, Windows). If you
have a standard directory structure within a GRASS_ADDONS_PATH (e.g., make
sure that path/bin is listed), it should make it easier to automatically
install such binaries.

4. Back to menus

You guys are working out all the hard stuff. But I'd like to go ahead and
get a menu parser built in TclTk AND wxPython to deal with whatever you come
up with.

As a first go around, I'd still like to keep it a flat menu if you don't
mind, so we can make it robust and reliable--and provide added value to your
extension system.

Thinking about it, more complex menu structure (submenus) should probably be
done in the menu file rather than in a directory structure for the extension
files themselves. The latter could get very complicated and messy. I don't
think submenus would be all that difficult to implement eventually, but
let's just start flat and see how it goes.

Can I expect to find the following?

1) a SINGLE ascii menu file, named xtnmenu.dat
2) that can be found in the directory specified by GRASS_ETC_PATH
3) in which each line contains: "main:<menu item name>:<executable
name>:<help text>" OR contains "main:separator::"

Note the term 'main' as the first line of a xtnmenu.dat file line. The idea
is that if we add submenus, we can replace 'main' by 'submenu',
'submenustart', or something along that line to do nested submenus.

If this is OK, I'll craft a parser in gmenu.tcl with some dummy entries for
you all to test.

Michael

__________________________________________
Michael Barton, Professor of Anthropology
Director of Graduate Studies
School of Human Evolution & Social Change
Center for Social Dynamics & Complexity
Arizona State University

phone: 480-965-6213
fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton

On Sep 22, 2007, at 11:17 AM, Michael Barton wrote:

Yes. I would be in favour of that: one plugin dir for each user and
nothing more (to worry about).

I don't know how g.manual currently searches for the main GRASS doc
directory, but it seems like it wouldn't be hard to change it to look in a
doc directory within GRASS_ADDON_PATH.

Like other extension files, this would survive a binary update on Mac and
Windows.

We should decide if we want keep GRASS_ADDON_PATH as the path(s) to bin/ and add GRASS_ADDON_HELP. Or simplify all the GRASS_ADDON_* to GRASS_ADDON_PATH which would be the base of a structured folder with bin/, docs/, lib/, etc/ subfolders.

I like the simplifying idea, but it could break users' shell inits (for setting their personal scripts dirs) until they reread the docs to see what's going on. And it would make using 6.2 and 6.3 together difficult for setting such psersonal scripts dirs.

Maybe a different name base would be best, like GRASS_XTN_PATH (matches the 'xtn' naming proposed for the menu file and what's currently used in the GEM menus). We could then keep GRASS_ADDON_PATH as a legacy variable in init.sh. GRASS_ADDON_ETC is so new that it could be dropped completely.

Also, extensions need to ship with different binaries for different
systems. We could try and guess the right ones automatically for the
user but I don't suppose there is a portable do of doing that.

Binaries would be highly desirable for many users when possible. But whether
or not they would get built should be up to the developer to hassle with. It
isn't realistic to try to create a binary for all possible systems, but a
limited number could cover a lot (e.g., RPM's, Debian, Mac, Windows). If you
have a standard directory structure within a GRASS_ADDONS_PATH (e.g., make
sure that path/bin is listed), it should make it easier to automatically
install such binaries.

Name the tarballs of any binaries with the platform's ARCH (or portion thereof) as specified in platform.make. A user can easily see which one they need from the name. Trying to figure out automatically if a binary is appropriate for a platform could be simple with this (but could be a bit tricky on OSX with multi-arch binaries and compatibility between major OS versions).

A simple script (or binary prog like GEM) can copy them to one of the bin/ folders in GRASS_ADDON_PATH (so make tools aren't needed).

Can I expect to find the following?

1) a SINGLE ascii menu file, named xtnmenu.dat
2) that can be found in the directory specified by GRASS_ETC_PATH
3) in which each line contains: "main:<menu item name>:<executable
name>:<help text>" OR contains "main:separator::"

or a comment. A structured comment(s) can be used to separate auto-generated menus from installers or startup from manually-added user menus (or a future menu editor), so that installers/startup routines can easily figure out what's theirs to manage. I believe GEM does this currently.

Note the term 'main' as the first line of a xtnmenu.dat file line. The idea
is that if we add submenus, we can replace 'main' by 'submenu',
'submenustart', or something along that line to do nested submenus.

If this is OK, I'll craft a parser in gmenu.tcl with some dummy entries for
you all to test.

Looks good.

-----
William Kyngesburye <kyngchaos*at*kyngchaos*dot*com>
http://www.kyngchaos.com/

"Mon Dieu! but they are all alike. Cheating, murdering, lying, fighting, and all for things that the beasts of the jungle would not deign to possess - money to purchase the effeminate pleasures of weaklings. And yet withal bound down by silly customs that make them slaves to their unhappy lot while firm in the belief that they be the lords of creation enjoying the only real pleasures of existence....

- the wisdom of Tarzan