[GRASS5] 5.7/6.0: some considerations

Hi,

due to the recent massive complaints about speed and scalability
of the GRASS 5.7-CVS data conversion modules (maybe other modules
as well) and about missing/incomplete documentation I would like
to post here a few considerations:

In 5.7 the vector engine was, based on the existing code, more or less
rewritten, almost by Radim. As spatial index, 3D faces and 3D kernel,
category index, multilayer, flexible database management system (SQL
based, with support of external databases and the new internal format)
were integrated, all algorithms of the vector library depending
modules had subsequently to be rewritten as well. Then, of course the
documentation as well (from scratch) as everything changed.
Along with a new Makefile system, new modules such as v.digit and d.m,
GUI windows generation at run-time, semiautomated HTML/MAN user
documentation, merge of the entire programmer's manual in doxygen
format into the source code, vector network analysis applications etc
etc a lot of development has happened since 2002.
In my opinion there is not so much to complain about.

Looking at the 5.7-CVS ChangeLog statistics, we see that 68%
of all contributions were submitted by only two developers (not
considering the recent 2500 code merge commits of Bernhard).

The current state of 5.7 probably reflects what's possible for such a
small number of developers, who, btw, are not supposed to work full
time for GRASS development. It's not that I intend to complain about
missing contributions from others. Valuable submissions have been
integrated the last two years. And GRASS could even survive with a
single developer and the related Web-infrastructure. But naturally
things happen much slower with a small number of contributors.
This was probably not very clear yet to everybody.

Personally, I'm also grateful to all the 5.7 users and power users who
continuously update (certainly annoying/risky from time to time) and
test the current developments. This is a valuable contribution as our
testing here at ITC-IRST (yes, we are using 5.7 in productional mode)
is certainly not sufficient to catch all possible combinations of
flags/parameters/projections/data types etc.

The strategy of the mentioned two developers who contributed most to
5.7 is to stabilize the existing code. However, to stabilize does not
necessarily mean to optimize for speed. Unless more people contribute
here, this will be postponed for 6.x with 6.x>6.0.

Main reason is that we simply have to get out a new stable version.
Otherwise nobody will start to package it and add it to Linux
distributions etc (see especially the discussions in Debian-GIS
mailing list). No packager is willing to invest time into a "moving
target" such as the CVS snapshots are. And very few "normal" users who
want a stable version for production consider to run GRASS CVS
(see the repeated questions on 5.0.3 which is still considered as the
only existing stable version in parts of the community).

As discussed several times in the last years, I still propose to
follow the "release often" paradigm rather than "let's postpone again"
(e.g. for a certain feature which then often doesn't happen to be
implemented). For example, the bugtracker is partially ignored by a
number of developers which suggests (to me) that we should not hold
6.0 as many of these old 5.0.x problems were already solved in the new
version. But: 6.0 must become accessible to the "normal" user.

And the group of 5.7 users who recently complained about various
problems are all power users/developers who are able to work from CVS.
Once 6.0 is out, we can (immediately, depends on contributions) go for
speed optimization etc. But for now, let's use our energy to clean up
existing stuff and add missing docs (also more translators are
urgently needed!); at least the above mentioned two developers think
so.

Looking at external projects such as QGIS and JAVAGRASS, we can say
that the new GRASS seems to be of major interest. But also the
developers of these projects will need a release to build on top of
that.

Again: IMHO there is not much to complain about. The upcoming version
comes with excellent features. The wider GRASS community is waiting
for it. Of course, don't hesitate to post improvements. They are more
than appreciated.

So far my remarks,

Markus

I agree. My hat is off to Markus and Radim for taking the lead in making
GRASS such a high quality piece of software. People are pretty amazed when I
show them what GRASS can do.

There are bugs and some annoying things that could be better. But I've
extensively used products from ESRI and MapInfo and have run into at least
as many (if not more) bugs, undocumented features/procedures, and annoying
incomprehensible behaviors in these commercial, and EXPENSIVE products as I
have in GRASS. At least on my Mac, GRASS is much more stable (i.e., no
freezes or crashes) than especially ESRI products under Windows. Just my 2
cents worth.

Michael

On 12/5/04 6:51 AM, "Markus Neteler" <neteler@itc.it> wrote:

Hi,

due to the recent massive complaints about speed and scalability
of the GRASS 5.7-CVS data conversion modules (maybe other modules
as well) and about missing/incomplete documentation I would like
to post here a few considerations:

In 5.7 the vector engine was, based on the existing code, more or less
rewritten, almost by Radim. As spatial index, 3D faces and 3D kernel,
category index, multilayer, flexible database management system (SQL
based, with support of external databases and the new internal format)
were integrated, all algorithms of the vector library depending
modules had subsequently to be rewritten as well. Then, of course the
documentation as well (from scratch) as everything changed.
Along with a new Makefile system, new modules such as v.digit and d.m,
GUI windows generation at run-time, semiautomated HTML/MAN user
documentation, merge of the entire programmer's manual in doxygen
format into the source code, vector network analysis applications etc
etc a lot of development has happened since 2002.
In my opinion there is not so much to complain about.

Looking at the 5.7-CVS ChangeLog statistics, we see that 68%
of all contributions were submitted by only two developers (not
considering the recent 2500 code merge commits of Bernhard).

The current state of 5.7 probably reflects what's possible for such a
small number of developers, who, btw, are not supposed to work full
time for GRASS development. It's not that I intend to complain about
missing contributions from others. Valuable submissions have been
integrated the last two years. And GRASS could even survive with a
single developer and the related Web-infrastructure. But naturally
things happen much slower with a small number of contributors.
This was probably not very clear yet to everybody.

Personally, I'm also grateful to all the 5.7 users and power users who
continuously update (certainly annoying/risky from time to time) and
test the current developments. This is a valuable contribution as our
testing here at ITC-IRST (yes, we are using 5.7 in productional mode)
is certainly not sufficient to catch all possible combinations of
flags/parameters/projections/data types etc.

The strategy of the mentioned two developers who contributed most to
5.7 is to stabilize the existing code. However, to stabilize does not
necessarily mean to optimize for speed. Unless more people contribute
here, this will be postponed for 6.x with 6.x>6.0.

Main reason is that we simply have to get out a new stable version.
Otherwise nobody will start to package it and add it to Linux
distributions etc (see especially the discussions in Debian-GIS
mailing list). No packager is willing to invest time into a "moving
target" such as the CVS snapshots are. And very few "normal" users who
want a stable version for production consider to run GRASS CVS
(see the repeated questions on 5.0.3 which is still considered as the
only existing stable version in parts of the community).

As discussed several times in the last years, I still propose to
follow the "release often" paradigm rather than "let's postpone again"
(e.g. for a certain feature which then often doesn't happen to be
implemented). For example, the bugtracker is partially ignored by a
number of developers which suggests (to me) that we should not hold
6.0 as many of these old 5.0.x problems were already solved in the new
version. But: 6.0 must become accessible to the "normal" user.

And the group of 5.7 users who recently complained about various
problems are all power users/developers who are able to work from CVS.
Once 6.0 is out, we can (immediately, depends on contributions) go for
speed optimization etc. But for now, let's use our energy to clean up
existing stuff and add missing docs (also more translators are
urgently needed!); at least the above mentioned two developers think
so.

Looking at external projects such as QGIS and JAVAGRASS, we can say
that the new GRASS seems to be of major interest. But also the
developers of these projects will need a release to build on top of
that.

Again: IMHO there is not much to complain about. The upcoming version
comes with excellent features. The wider GRASS community is waiting
for it. Of course, don't hesitate to post improvements. They are more
than appreciated.

So far my remarks,

Markus

____________________
C. Michael Barton, Professor of Anthropology
School of Human Evolution and Social Change
PO Box 872402
Arizona State University
Tempe, AZ 85287-2402
USA

Phone: 480-965-6262
Fax: 480-965-7671
www: <www.public.asu.edu/~cmbarton>

Markus Neteler wrote:

The strategy of the mentioned two developers who contributed most to
5.7 is to stabilize the existing code. However, to stabilize does not
necessarily mean to optimize for speed. Unless more people contribute
here, this will be postponed for 6.x with 6.x>6.0.

Markus,

those "speed complaints" are just references to a possible serious bug. No one says the work that has been done is bad. In such major code revisions that can happen. We were testing grass57 and found a problem. I was just wondering if this is an inherent feature of the new vector engine or something else. If this is a bug, it should be fixed.
I was out for a couple of days, sorry if I missed something in this discussion.

Jaro

-=x=-
Skontrolované antivírovým programom NOD32

On Wed, Dec 08, 2004 at 09:56:58AM +0100, Jaro Hofierka wrote:

Markus Neteler wrote:

>
>The strategy of the mentioned two developers who contributed most to
>5.7 is to stabilize the existing code. However, to stabilize does not
>necessarily mean to optimize for speed. Unless more people contribute
>here, this will be postponed for 6.x with 6.x>6.0.

Markus,

those "speed complaints" are just references to a possible serious bug.
No one says the work that has been done is bad. In such major code
revisions that can happen. We were testing grass57 and found a problem.
I was just wondering if this is an inherent feature of the new vector
engine or something else. If this is a bug, it should be fixed.
I was out for a couple of days, sorry if I missed something in this
discussion.

Jaro,

sure, bugs must be fixed. But more developers are needed to do so.
I don't think that slowliness is a new intended feature of the new
vector engine. But due to the very limited number of contributors
to the new vector engine stability/correctness with come first, then
speed.
If it is a memory leak it should be fixed. I played with 'valgrind'
but didn't get a clue where to start. Since we have skilled people
in the development team I still have the hope that some of these
developers tries as well.

Also I try to keep pace with the zillion new reports we currently
receive, being a sort of garbage collector to solve the many minor
issues. That's all I can provide.

Markus

Jaro Hofierka wrote:

Markus,

those "speed complaints" are just references to a possible serious bug. No one says the work that has been done is bad. In such major code revisions that can happen. We were testing grass57 and found a problem. I was just wondering if this is an inherent feature of the new vector engine or something else. If this is a bug, it should be fixed.
I was out for a couple of days, sorry if I missed something in this discussion.

Jaro

The system is more complex, with more functions and speed is the price.
I cannot say where precisely is the biggest problem, I asked few days ago for simple benchmark regarding the speed, just to remind:

> Can you post here times consumed by?:
> s.in.ascii
> v.in.ascii -t
> v.in.ascii / dbf
> v.in.ascii / postgres
> insert attributes using insert statements + db.execute (in one step)
> / dbf
> / postgres
> v.in.sites / dbf
> v.in.sites / postgres
> v.surf.rst layer=0 (i.e. z coor)
> v.surf.rst / dbf
> v.surf.rst / postgres
> s.surf.rst (5.4)

but until now, I have not recieved any results, so I take it, that people are not interested in solving this problem.

I guess that most time is spend in attributes handling. Probably the file+DB architecture is wrong. DB is not suitable for random access to large data sets from applications. Data manipulation via SQL for large datasets is another problem. Either files only or DB only, the combination is bad.
It can be improved a lot, I believe, for DBF driver, using fuctions instead of SQL between modules and the driver.
Some embeded database is also desirable.

It is just work which should be done and which nobody does.

I am always surprised again and again why people complain about something missing instead of posting contributions. Maybe it is not quite clear, but GRASS is free software now. Irst is not CERL. GRASS is like the GRASS comunity, if the comunity is poor, GRASS will be poor.
It seems that 99.9% of users believe that there is a large group of developers waiting for user wishes, keen on implementing them.
That is not true, if you need something in GRASS, usually you have to write it yourself or pay somebody. That is principle of free software development. Sorry, no free beer.

Radim

Thank you Radim, you do a socko job!

Drop me a mail with your post adress, and I will send you and Markus some gorgeous-brain-wiping Bavarian Beer to honour your efforts.

Beside, I understood the questions asked to the list not at all as a criticims but more as a call for help.

Mit freundlichen Grüßen / With kindest regards

Stefan Paulick

http://www.urbeli.com
mailto://stefan.paulick@urbeli.com
/*----------------------*/

Radim Blazek schrieb:

I am always surprised again and again why people complain about something missing instead of posting contributions. Maybe it is not quite clear, but GRASS is free software now. Irst is not CERL. GRASS is like the GRASS comunity, if the comunity is poor, GRASS will be poor.
It seems that 99.9% of users believe that there is a large group of developers waiting for user wishes, keen on implementing them.
That is not true, if you need something in GRASS, usually you have to write it yourself or pay somebody. That is principle of free software development. Sorry, no free beer.

Radim

Mit freundlichen Grüßen / With kindest regards

Stefan Paulick

http://www.urbeli.com
mailto://stefan.paulick@urbeli.com
/*----------------------*/

Thanks, but I wanted to say: until GRASS users recognize that sending just thanks and beer is not enough, GRASS will be what it is - merda.

Radim

Stefan Paulick wrote:

Thank you Radim, you do a socko job!
Drop me a mail with your post adress, and I will send you and Markus some gorgeous-brain-wiping Bavarian Beer to honour your efforts.
Beside, I understood the questions asked to the list not at all as a criticims but more as a call for help.

Mit freundlichen Grüßen / With kindest regards
Stefan Paulick

http://www.urbeli.com
mailto://stefan.paulick@urbeli.com
/*----------------------*/

Radim Blazek schrieb:

I am always surprised again and again why people complain about something missing instead of posting contributions. Maybe it is not quite clear, but GRASS is free software now. Irst is not CERL. GRASS is like the GRASS comunity, if the comunity is poor, GRASS will be poor.
It seems that 99.9% of users believe that there is a large group of developers waiting for user wishes, keen on implementing them.
That is not true, if you need something in GRASS, usually you have to write it yourself or pay somebody. That is principle of free software development. Sorry, no free beer.
Radim

Mit freundlichen Grüßen / With kindest regards
Stefan Paulick

http://www.urbeli.com
mailto://stefan.paulick@urbeli.com
/*----------------------*/

On Thursday 09 December 2004 13:57, Radim Blazek wrote:

Thanks, but I wanted to say: until GRASS users recognize that sending
just thanks and beer is not enough, GRASS will be what it is - merda.

well as the GRASS survey revealed (results will be published soon!) there are
a lot of users who want to contribute but do not know how - hence the user
community already recognised that they have to help but it takes time until
they managed to get "jobs" to do e.g. via the "to do" wiki section.

just to encourage the developers that helping hands are on their way :wink:
Martin

Stefan Paulick wrote:
> Thank you Radim, you do a socko job!
> Drop me a mail with your post adress, and I will send you and Markus
> some gorgeous-brain-wiping Bavarian Beer to honour your efforts.
> Beside, I understood the questions asked to the list not at all as a
> criticims but more as a call for help.
>
> Mit freundlichen Gr�゚en / With kindest regards
> Stefan Paulick
>
> http://www.urbeli.com
> mailto://stefan.paulick@urbeli.com
> /*----------------------*/
>
> Radim Blazek schrieb:
>> I am always surprised again and again why people complain about
>> something missing instead of posting contributions. Maybe it is not
>> quite clear, but GRASS is free software now. Irst is not CERL. GRASS
>> is like the GRASS comunity, if the comunity is poor, GRASS will be
>> poor. It seems that 99.9% of users believe that there is a large group
>> of developers waiting for user wishes, keen on implementing them.
>> That is not true, if you need something in GRASS, usually you have to
>> write it yourself or pay somebody. That is principle of free software
>> development. Sorry, no free beer.
>> Radim
>
> Mit freundlichen Gr�゚en / With kindest regards
> Stefan Paulick
>
> http://www.urbeli.com
> mailto://stefan.paulick@urbeli.com
> /*----------------------*/

_______________________________________________
grass5 mailing list
grass5@grass.itc.it
http://grass.itc.it/mailman/listinfo/grass5

On Thu, Dec 09, 2004 at 03:03:22PM +0100, Martin Wegmann wrote:

On Thursday 09 December 2004 13:57, Radim Blazek wrote:
> Thanks, but I wanted to say: until GRASS users recognize that sending
> just thanks and beer is not enough, GRASS will be what it is - merda.

well as the GRASS survey revealed (results will be published soon!) there are
a lot of users who want to contribute but do not know how - hence the user
community already recognised that they have to help but it takes time until
they managed to get "jobs" to do e.g. via the "to do" wiki section.

Well, there are quite a few (known) options:

http://grass.itc.it/
  -> Get involved (right menu)

just to encourage the developers that helping hands are on their way :wink:

Cool.

Martin

Markus