[GRASS-dev] Some ideas about future GRASS development

GRASS GRASS developer
1

Hi all!

That rather radical ideas I present here are rather for future, at least for GRASS 8, but I'd like present it now for long-term reflection.

Probably all notice that for over two years there is big increase in add-on repository (including me). There are modules of different quality: from fully GRASS toolsets, to shell or python scripts, from actively developed tools to abandoned, from all-purpose tools to very specialized etc. I also think that that activity will be grown due to substitute shell script by python

Similar situation is in main GRASS branch: there are modules for all like conversion tools, interpolation methods, georeferencing etc, and very specialized modules for very limited group of users (like wild fire), there are also some modules out of date.

I'm not enthusiastic about moving new modules into main branch. Almost every module has different coding style and it will lasting in future that GRASS would be difficult to maintain. On the other hand some people complains that some interesting modules are only available as add-ons (I assume for some reasons they cannot install it)

So my suggestion is to rearrange future GRASS form two layers (main branch/add-on) into three layers architecture:

1) GRASS core layer: much limited limited than now, only GIS environment and basic, all-puropse tools, slow changes, great stability
2) GRASS toolset layer: oficcial GRASS thematic tools and toolsets (like terrain analysis, hydrological analysis, photo-interpretation, landscape analysis etc,) every toolset with its maintainer, rapid development, new ready to use tools after quality control may appear here, also some of current main branch tool shall be moved to that layer
3) GRASS community layer: everything else like experimental, actively development new tools, that what do not pass quality control, simple scripts, etc....

What benefits:
for developers and contributors: much clear situation and better publication path.
Toolset layer should be much more open for new tools than current GRASS main branch

for users: faster access to new tools.
There is no doubt that new tools are faster developed (less risk) than GRASS core
Binaries with toolsets could be maintained as separate apt/urpmi/pacman/yum/exe etc packages, so it may appear in linux repository separetly form GRASS core.

There is only loose ideas. Most of them are of course taken from R (core/toolsets/rest of packages; separate core and package development) but I think it is worth of some discuss ...

regards
Jarek

2

Hi Jarek,

2010/4/28 Jarosław Jasiewicz <jarekj@amu.edu.pl>:

That rather radical ideas I present here are rather for future, at least for
GRASS 8, but I'd like present it now for long-term reflection.

Probably all notice that for over two years there is big increase in add-on
repository (including me). There are modules of different quality: from
fully GRASS toolsets, to shell or python scripts, from actively developed
tools to abandoned, from all-purpose tools to very specialized etc. I also
think that that activity will be grown due to substitute shell script by
python

Similar situation is in main GRASS branch: there are modules for all like
conversion tools, interpolation methods, georeferencing etc, and very
specialized modules for very limited group of users (like wild fire), there
are also some modules out of date.

I'm not enthusiastic about moving new modules into main branch. Almost every
module has different coding style and it will lasting in future that GRASS
would be difficult to maintain. On the other hand some people complains that
some interesting modules are only available as add-ons (I assume for some
reasons they cannot install it)

So my suggestion is to rearrange future GRASS form two layers (main
branch/add-on) into three layers architecture:

1) GRASS core layer: much limited limited than now, only GIS environment and
basic, all-puropse tools, slow changes, great stability
2) GRASS toolset layer: oficcial GRASS thematic tools and toolsets (like
terrain analysis, hydrological analysis, photo-interpretation, landscape
analysis etc,) every toolset with its maintainer, rapid development, new
ready to use tools after quality control may appear here, also some of
current main branch tool shall be moved to that layer
3) GRASS community layer: everything else like experimental, actively
development new tools, that what do not pass quality control, simple
scripts, etc....

What benefits:
for developers and contributors: much clear situation and better publication
path.
Toolset layer should be much more open for new tools than current GRASS main
branch

for users: faster access to new tools.
There is no doubt that new tools are faster developed (less risk) than GRASS
core
Binaries with toolsets could be maintained as separate
apt/urpmi/pacman/yum/exe etc packages, so it may appear in linux repository
separetly form GRASS core.

I like your idea in general. I think there is enough time to discuss
new repository layout also for GRASS 7. I have created for this
purpose a wiki page [1].

Cheers, Martin

[1] http://grass.osgeo.org/wiki/GRASS_repository_layout_proposal

--
Martin Landa <landa.martin gmail.com> * http://gama.fsv.cvut.cz/~landa

3

discussion is useful; I've added my comments to the wiki's Talk page.
maybe not surprisingly I've started with the cautious approach ...

cc'd:

fwiw I'm not in favour of the R CRAN approach for GRASS.

    * One of our best selling points vs. the competition is that you don't have to buy expensive addon toolboxes to have it do what you want to do.
    * It makes it a lot harder for new users to get started with what they want to do. Even when done well it's a brittle system dependent on 100% uptime servers etc. which in practice do not exist.
    * Non-"core" modules will be neglected by the core devs and die from bit rot. (outside of grep's reach)
    * Those "non-core" modules have personally led me into all new ideas and directions outside of my normal field of study, which has rather positively affected the direction of my career and let me solve problems in novel ways for my peers that only cross-discipline tools/perspective could introduce us to.
    * Our download size is only about 25ish megs. that's tiny. Docs are bigger than code. Windows deps "aren't our fault" and switching to a different distribution model won't help that much at all.
    * Rather than focus development I fear it will dilute it. Divided we fall..
    * Big change is big work which could more productively be funneled into more critical pursuits. (I am not against needed change, but very against change-for-change's-sake.)

regards,
Hamish

4

Hi,

2010/5/10 Hamish <hamish_b@yahoo.com>:

* One of our best selling points vs. the competition is that you don't have to buy expensive addon toolboxes to have it do what you want to do.
* It makes it a lot harder for new users to get started with what they want to do. Even when done well it's a brittle system dependent on 100% uptime servers etc. which in practice do not exist.

advanced Tools Manager could help with that, see the minimalistic approach [1].

* Non-"core" modules will be neglected by the core devs and die from bit rot. (outside of grep's reach)
* Those "non-core" modules have personally led me into all new ideas and directions outside of my normal field of study, which has rather positively affected the direction of my career and let me solve problems in novel ways for my peers that only cross-discipline tools/perspective could introduce us to.

I am not afraid of that. Core should be minimalistic, the most modules
(e.g. v.net.*, etc) could be moved to grass-tools. Most of them will
be maintain in the same manner as now, they just live in separate
repo.

* Our download size is only about 25ish megs. that's tiny. Docs are bigger than code. Windows deps "aren't our fault" and switching to a different distribution model won't help that much at all.
* Rather than focus development I fear it will dilute it. Divided we fall..
* Big change is big work which could more productively be funneled into more critical pursuits. (I am not against needed change, but very against change-for-change's-sake.)

The main question is which modules should to moved from grass-addons
to trunk. Who will decide, PSC? As reference I can mention modules
from GIPE which has been moved to trunk few months ago.

Basically, I like the idea of minimalistic repo for libs and core
modules which will be maintained carefully to be very stable. And
grass-tools with solid and *maintained* modules. The rest in
grass-addons. Currently grass-addons contains a mess, you can find
very good modules but also unusable rubbish. The user is lost.

Martin

[1] http://grass.osgeo.org/wiki/WxGUI#Extension_Manager

--
Martin Landa <landa.martin gmail.com> * http://gama.fsv.cvut.cz/~landa

5

Hi,

2010/5/10 Hamish <hamish_b@yahoo.com>:

* Our download size is only about 25ish megs. that's tiny. Docs are bigger than code. Windows deps "aren't our fault" and switching to a different distribution model won't help that much at all.

well, then we can build separate packages based on the tools included,
at least 'grass-core', grass-full`.

Martin

--
Martin Landa <landa.martin gmail.com> * http://gama.fsv.cvut.cz/~landa

6

Hamish:

> * Our download size is only about 25ish megs.
> that's tiny. Docs are bigger than code. Windows deps "aren't
> our fault" and switching to a different distribution model
> won't help that much at all.

Martin:

well, then we can build separate packages based on the
tools included, at least 'grass-core', grass-full`.

my point is that splitting it up doesn't gain us much space-wise. An extra
50 modules might only cost another 6mb on top of the core distro.

For WinGrass 80mb+6mb vs 86mb. might as well just ship the full 86mb
version in the first place and save everyone the extra 1000% installation
trouble.

(mean size of modules in bin/ for me is 60k .... +50 modules *60k = 3mb)

as far as addon quality variability goes, I don't see that changing much
regardless of the system. As it is a technical decision about what is
up-to-quality so I prefer to leave the PSC out of it. FWIW my general
devel mode for new (mostly quick script) modules is to develop them in
addons, and once they are of what I consider to be release quality if they
have general appeal move them into main. Others which are very useful to
me but a bit less general use or only 95% happy with I've left in addons
to avoid cluttering the main tree. I figure anything from addons useful
enough will generally generate enough feedback and core-dev interest to
naturally migrate to the main tree. If it has a champion from the core
dev team with write access to the core development tree who cares enough
to move it across, then it has a long-term maintainer and is not much
extra load to the other core devs. :slight_smile:

mostly I just want to be convinced that the existing model is badly
broken before we try and replace it by something which may or may not
be the same or better.

regards,
Hamish

ps- wrt GIPE, any thoughts on renaming r.vi to the less jargony r.vegindex,
and should i.sunhours be r.sunhours, etc? Still some QA work to do on all
newly introduced modules.. (nothing really badly wrong about them, just
that some of their peer-modules have 20 years of field testing behind
them, so our standards are very high and consistent quality takes more
effort than we might think)

on the subject of fighting jargon (and moving totally off-topic :), I had
spent some hours going through the GUI menus removing jargon which is
natural to us long-time users but totally meaningless to new users. (e.g.
how should they know that "GDAL" will open their file when it is a ".tif"
not a ".gdal") but all that got clobbered. I am unsure of the automated
wx menu sync tool, is it and a guide as to how we should work with the
menus documented anywhere? anyway I feel that menu items should be a
little more "easy" and module descriptions be more precise/accurate as to
function. using the text same for both while easier isn't always very
appropriate.

7

Hi,

2010/5/10 Hamish <hamish_b@yahoo.com>:

well, then we can build separate packages based on the
tools included, at least 'grass-core', grass-full`.

my point is that splitting it up doesn't gain us much space-wise. An extra
50 modules might only cost another 6mb on top of the core distro.
For WinGrass 80mb+6mb vs 86mb. might as well just ship the full 86mb
version in the first place and save everyone the extra 1000% installation
trouble.

(mean size of modules in bin/ for me is 60k .... +50 modules *60k = 3mb)

OK, anyway from my point of view space-size is not the reason of this
changes. The main reason for me is to clean up the GRASS modules and
to split them to 'core', 'tools' (well maintained) and 'addons' for
rest.

devel mode for new (mostly quick script) modules is to develop them in
addons, and once they are of what I consider to be release quality if they
have general appeal move them into main. Others which are very useful to
me but a bit less general use or only 95% happy with I've left in addons
to avoid cluttering the main tree. I figure anything from addons useful
enough will generally generate enough feedback and core-dev interest to
naturally migrate to the main tree. If it has a champion from the core

well, the main tree would be nice too keep minimalistic. Then is less
work to maintain it.

Martin

[...]

not a ".gdal") but all that got clobbered. I am unsure of the automated
wx menu sync tool, is it and a guide as to how we should work with the
menus documented anywhere? anyway I feel that menu items should be a
little more "easy" and module descriptions be more precise/accurate as to
function. using the text same for both while easier isn't always very
appropriate.

wxGUI menu could be easily by g.extension when installing new
module(s). Of course the proposed repo layout requires solid
g.extension module.

--
Martin Landa <landa.martin gmail.com> * http://gama.fsv.cvut.cz/~landa

8

Hi Martin, Hamish, all...

Hi,

2010/5/10 Hamish <hamish_b@yahoo.com>:
  

   * One of our best selling points vs. the competition is that you don't have to buy expensive addon toolboxes to have it do what you want to do.
   * It makes it a lot harder for new users to get started with what they want to do. Even when done well it's a brittle system dependent on 100% uptime servers etc. which in practice do not exist.
    

Now new user is frequently referred to add-ons where are modules of different quality. And new users are really lost. Now everyone can add something to add-on nobody control its quality or sense. Toolboxes (or any similar) will give quality control on new modules. Everything in toolboxes is good quality, but may not be useful for everyone.

advanced Tools Manager could help with that, see the minimalistic approach [1].
  

Even not very advanced.

   * Non-"core" modules will be neglected by the core devs and die from bit rot. (outside of grep's reach)
   * Those "non-core" modules have personally led me into all new ideas and directions outside of my normal field of study, which has rather positively affected the direction of my career and let me solve problems in novel ways for my peers that only cross-discipline tools/perspective could introduce us to.
    

I think both core and toolboxes could be still "GRASS-CORE" and will be maintained together (if require), in the same manner.

I am not afraid of that. Core should be minimalistic, the most modules
  

   * Our download size is only about 25ish megs. that's tiny. Docs are bigger than code. Windows deps "aren't our fault" and switching to a different distribution model won't help that much at all.

   * Rather than focus development I fear it will dilute it. Divided we fall..
    

This is not idea of divide! This is idea of better integration contributors and developers. I't is good you point out dangerous of such approach but I is idea of increase the number of developers with different competitions.

   * Big change is big work which could more productively be funneled into more critical pursuits. (I am not against needed change, but very against change-for-change's-sake.)
    

So you think we do not need new functionalities in GRASS?

The main question is which modules should to moved from grass-addons
to trunk. Who will decide, PSC? As reference I can mention modules
from GIPE which has been moved to trunk few months ago.

Basically, I like the idea of minimalistic repo for libs and core
modules which will be maintained carefully to be very stable. And
grass-tools with solid and *maintained* modules. The rest in
grass-addons. Currently grass-addons contains a mess, you can find
very good modules but also unusable rubbish. The user is lost.
  

it is exactly my idea, but I think it is problem of feature, I think problem is, but I didn't say that my proposition will really help. In general the aim is to join more people into GRASS developing, not divide GRASS into parts. In some parts GRASS shall develop slow and concentrate on "critical pursuits" in some parts (toolboxes) it could develop faster - nothing more...
best
Jarek

9

Martin Landa pisze:

Hi,

2010/5/10 Hamish <hamish_b@yahoo.com>:
  

well, then we can build separate packages based on the
tools included, at least 'grass-core', grass-full`.
      

my point is that splitting it up doesn't gain us much space-wise. An extra
50 modules might only cost another 6mb on top of the core distro.
For WinGrass 80mb+6mb vs 86mb. might as well just ship the full 86mb
version in the first place and save everyone the extra 1000% installation
trouble.

(mean size of modules in bin/ for me is 60k .... +50 modules *60k = 3mb)
    
OK, anyway from my point of view space-size is not the reason of this
changes. The main reason for me is to clean up the GRASS modules and
to split them to 'core', 'tools' (well maintained) and 'addons' for
rest.

devel mode for new (mostly quick script)

chmm.. I quick look into add-on repository and most new modules from 2009/10 are rather C-modules than quick scripts....

modules is to develop them in
addons, and once they are of what I consider to be release quality if they
have general appeal move them into main. Others which are very useful to
me but a bit less general use or only 95% happy with I've left in addons
to avoid cluttering the main tree.
I figure anything from addons useful
enough will generally generate enough feedback and core-dev interest to
naturally migrate to the main tree. If it has a champion from the core
    
well, the main tree would be nice too keep minimalistic. Then is less
work to maintain it.

Martin

[...]

not a ".gdal") but all that got clobbered. I am unsure of the automated
wx menu sync tool, is it and a guide as to how we should work with the
menus documented anywhere? anyway I feel that menu items should be a
little more "easy" and module descriptions be more precise/accurate as to
function. using the text same for both while easier isn't always very
appropriate.
    
wxGUI menu could be easily by g.extension when installing new
module(s). Of course the proposed repo layout requires solid
g.extension module.

10

On Monday 10 May 2010, Hamish wrote:

Hamish:
> > * Our download size is only about 25ish megs.
> > that's tiny. Docs are bigger than code. Windows deps "aren't
> > our fault" and switching to a different distribution model
> > won't help that much at all.

Martin:
> well, then we can build separate packages based on the
> tools included, at least 'grass-core', grass-full`.

my point is that splitting it up doesn't gain us much space-wise. An extra
50 modules might only cost another 6mb on top of the core distro.

For WinGrass 80mb+6mb vs 86mb. might as well just ship the full 86mb
version in the first place and save everyone the extra 1000% installation
trouble.

(mean size of modules in bin/ for me is 60k .... +50 modules *60k = 3mb)

Hi,

I can see the logic in splitting grass up into chunks to facilitate the
development of new modules without exposing the core modules to irresponsible
tinkering. I also like the way in which R is setup along these lines.
However, I don't think that there are enough willing individuals (as compared
to what is required to run something like CRAN) nor massive inflow of new
modules (as compared to R development) to warrant such a drastic change.
Furthermore, we don't even have the infrastructure to support such a system.
Maybe in the future, but not right now-- all efforts should be placed on
making GRASS 65 rock solid and GRASS 7 closer to a reality.

as far as addon quality variability goes, I don't see that changing much
regardless of the system. As it is a technical decision about what is
up-to-quality so I prefer to leave the PSC out of it. FWIW my general
devel mode for new (mostly quick script) modules is to develop them in
addons, and once they are of what I consider to be release quality if they
have general appeal move them into main. Others which are very useful to
me but a bit less general use or only 95% happy with I've left in addons
to avoid cluttering the main tree. I figure anything from addons useful
enough will generally generate enough feedback and core-dev interest to
naturally migrate to the main tree. If it has a champion from the core
dev team with write access to the core development tree who cares enough
to move it across, then it has a long-term maintainer and is not much
extra load to the other core devs. :slight_smile:

I'll second that- there is a wide range of stuff in the addons branch. I have
seen some very good stuff in addons that I would like to see in the GRASS
proper. I don't know how it would happen, but at some point we should do some
house cleaning in the addons repository. Perhaps we can get a couple of
interested devs + users to go through and rate each of the modules-- then
those scoring above a certain threshold would be moved into GRASS proper.

Recently sumitting a package to CRAN makes me realize just how much back-end
work, planning, and volunteer work goes into that system.

Good discussion,

Dylan

mostly I just want to be convinced that the existing model is badly
broken before we try and replace it by something which may or may not
be the same or better.

regards,
Hamish

ps- wrt GIPE, any thoughts on renaming r.vi to the less jargony r.vegindex,
and should i.sunhours be r.sunhours, etc? Still some QA work to do on all
newly introduced modules.. (nothing really badly wrong about them, just
that some of their peer-modules have 20 years of field testing behind
them, so our standards are very high and consistent quality takes more
effort than we might think)

on the subject of fighting jargon (and moving totally off-topic :), I had
spent some hours going through the GUI menus removing jargon which is
natural to us long-time users but totally meaningless to new users. (e.g.
how should they know that "GDAL" will open their file when it is a ".tif"
not a ".gdal") but all that got clobbered. I am unsure of the automated
wx menu sync tool, is it and a guide as to how we should work with the
menus documented anywhere? anyway I feel that menu items should be a
little more "easy" and module descriptions be more precise/accurate as to
function. using the text same for both while easier isn't always very
appropriate.

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

--
Dylan Beaudette
Soil Resource Laboratory
http://casoilresource.lawr.ucdavis.edu/
University of California at Davis
530.754.7341