Dear Nyall and all,
here are some of my thoughts mostly relating to GRASS GIS (although it may not express views of the whole GRASS GIS developer team).
On Thu, Feb 8, 2018 at 8:55 PM, Nyall Dawson <nyall.dawson@gmail.com> wrote:
Here’s the situation as I see it:
The past
We (the QGIS project) have struggled to keep a bunch of providers
stable and available with the base QGIS install, and the current
maintainers (Alex and Victor, others) have (understandably) struggled
with the burden of maintaining these alone and the time commitment
required to do so. Users have been frustrated by breakage which occurs
when the algorithms they depend on break due to changes in underlying
libraries for which the processing providers have not been adapted.
Despite this, I’d say overall it’s worked OK-ish up to now, but
definitely with lots of room for improvement.
The goals
I think everyone involved is in agreement that processing’s strength
comes when there’s many providers available and it’s able to tie
together processes regardless of which provider or library they come
from. So I’d say our common goals are:
- Encourage development of as many processing providers as possible
- Make it easy for developers to create and maintain these providers
- Make it easy for users to be able to use the providers
- Make everything stable and painfree for users
Any disagreements? No? Good 
So if we agree that those are the end goals, we should be using them
to drive this discussion.
The questions
The open, debatable questions are:
- Which is the best approach for allowing easy maintenance of
providers (regardless of whether the provider is maintained by the
core QGIS team or a 3rd party)? Is it keeping the code in the master
codebase and locking to QGIS releases, or publishing via plugins and
having independent release schedules? Which approach will encourage
more active maintenance of these providers and share the burden of
this maintenance?
One thing is where to keep the code, however I’m not sure what code are we talking about and I would like to talk about this first. I think that recent post by Moritz is bringing this up as well:
https://lists.osgeo.org/pipermail/grass-dev/2018-February/087420.html
As far as I know, QGIS Processing has a text file for each GRASS module describing its interface and maintenance of these takes time. However, every GRASS module can tell about itself when called with --interface-description parameter. If this is used, individual parameters don’t need to be maintained and any, even user provided module can be executed in processing.
The --interface-description parameter gives XML which would need to be converted or a new parameter, let’s say --qgis-description, would need to be added for a QGIS-specific format, there is for example --script for GRASS GIS interface description for scripts. --qgis-description would need to be in GRASS GIS code base while the conversion can be anywhere. See emails from Rashad discussing a possible implementation with the --qgis-description way and the OTB way:
https://lists.osgeo.org/pipermail/grass-dev/2018-February/087362.html
https://lists.osgeo.org/pipermail/grass-dev/2018-February/087390.html
The was discussed in the past and in fact, it is used in the QGIS GRASS plugin as Radim explains in this older email:
https://lists.osgeo.org/pipermail/grass-dev/2015-March/074629.html
However, it is not using the --interface-description XML only, but it is also using the qgm files to supply some additional information which means that this system still requires updates which are separate from the updates of GRASS modules. This can be avoided if the necessary information is updated upstream or sometimes even GRASS --interface-description format or the individual module descriptions are extended to include that what is needed by QGIS Processing. Here is a thread which disuses this issues although diverges into the following:
https://lists.osgeo.org/pipermail/grass-dev/2015-September/076138.html
One issue which is creating differences between QGIS Processing representation of the module and the GRASS one is the issue of advanced parameters. GRASS itself has mechanism to organize the parameters into groups and marks the required ones. This is of course something which is constantly being refined and it can be used and changed also for QGIS Processing as discussed in this 2015 post:
https://lists.osgeo.org/pipermail/grass-dev/2015-December/078000.html
Other issues besides the interface include list of GRASS modules usable and unusable in QGIS Processing, thematic tree of these modules, splitting modules and more. Many of these are discussed in this recent post:
https://lists.osgeo.org/pipermail/grass-dev/2018-February/087388.html
One of the issues is issue of formats (or import & export). Here again, it seems to that it is more advantageous to have a function in GRASS code base to care take of the import and, if needed, push the changes upstream to it (from QGIS point of view). Here is a related example which is a comment from Markus Neteler (regarding currently preferred solution for vector import) which would be part of PR/patch review process if the import function was originally submitted as extension of GRASS code base:
https://github.com/qgis/QGIS/pull/5426#issuecomment-345067457
Then there is of course the issue of GRASS modules which are more difficult to wrap in QGIS Processing because the inputs and outputs are not explicitly stated, most prominent example is probably r.mapcalc but there is couple of more, although we were able to reduce this number lately:
https://lists.osgeo.org/pipermail/grass-dev/2015-January/072979.html
See also the following post from Giovanni Manghi (sent off-list to Markus Neteler and shared by him) and a ticket commend by me for more examples and details:
https://lists.osgeo.org/pipermail/grass-dev/2014-August/070342.html
https://trac.osgeo.org/grass/ticket/2409#comment:17
Here is a good r.mapcalc-related example of (at least) asking for changes upstream (although sent privately to Markus Neteler who opened the issue):
https://trac.osgeo.org/grass/ticket/3431
- Who should be responsible for maintaining the providers? Should
maintenance be done by the QGIS core developer team or by 3rd parties?
I’m not sure if we can really separate this from the place in the code, but if there is a notion of pushing changes upstream then that gives definitively us more flexibility.
Besides maintenance, I think the initial development, which is necessary for a realistic maintenance plan, is a major issue. I, and I think some other GRASS devs too, would be happy to work on it if the financing is addressed. A different suggestion is here as well and that’s GSoC idea proposed by Martin Landa:
https://trac.osgeo.org/grass/wiki/GSoC/2018#ImproveGRASSintegrationinQGIS3
- Is it a core requirement that 3rd party providers are installed with
EVERY QGIS install, or is it acceptable to require interested users to
manually install the tools which they find useful? If the second
option is preferred, then how can we ensure that users are aware of
the availability of these tools?
If they install manually, on Windows, users can have the latest version of the software instead of the one packaged with QGIS. OSGeo4W is a different case; not sure about Mac. However, on Linux, the decision is up to the packagers, so in any case, the QGIS part needs to be able to work with different versions. Regardless of QGIS packaging efforts, I have seen many users with old GRASS who installed new standalone GRASS on the side, even recently.
On the other hand, many users benefit from the one package, because they need to get software approved, if they get QGIS packaged with GRASS approve, they got GRASS as well. Getting approval for each individual provider is likely more paperwork.
I hope this will help and let me know what I’m missing.
Best,
Vaclav
