[GRASS-dev] External providers in QGIS

GRASS GRASS developer
, , ,
1

Hi all,
following the discussion on qgis-dev ML:
https://lists.osgeo.org/pipermail/qgis-developer/2018-January/051701.html
it has been proposed to remove all external providers (namely OTB,
already removed, GRASS, and SAGA) from Processing in QGIS2 (GDAL will
remain as QGIS cannot be built without). This raised a number of
concerns, so PSC decided not to rush removing them from the upcoming 3.0
release, and to open a wider discussions about this, involving all the
interested parties, to find an optimal solution.
If volunteers outside QGIS core team, ideally from the respective
backends, could take the maintenance of these providers, not much would
change for the end user. If not, this will mean effectively missing
hundreds of algs from QGIS.
I personally propose to reinstate the OTB plugin with Rashad (from OTB)
as maintainer.
QGIS.ORG will seek to support any decision made with funding where possible.
Looking forward for your feedback.
All the best.
--
Paolo Cavallini - www.faunalia.eu
QGIS & PostGIS courses: http://www.faunalia.eu/training.html
https://www.google.com/trends/explore?date=all&geo=IT&q=qgis,arcgis

2

Hi Paolo,

pcav wrote

Hi all,
following the discussion on qgis-dev ML:
https://lists.osgeo.org/pipermail/qgis-developer/2018-January/051701.html
it has been proposed to remove all external providers (namely OTB,
already removed, GRASS, and SAGA) from Processing in QGIS2 (GDAL will
remain as QGIS cannot be built without). This raised a number of
concerns, so PSC decided not to rush removing them from the upcoming 3.0
release, and to open a wider discussions about this, involving all the
interested parties, to find an optimal solution.
If volunteers outside QGIS core team, ideally from the respective
backends, could take the maintenance of these providers, not much would
change for the end user. If not, this will mean effectively missing
hundreds of algs from QGIS.
I personally propose to reinstate the OTB plugin with Rashad (from OTB)
as maintainer.
QGIS.ORG will seek to support any decision made with funding where
possible.
Looking forward for your feedback.
All the best.
--
Paolo Cavallini

thanks for the follow up on this discussion.

Is there already a concrete technical documention of requirements, methods,
API, interface etc by QGIS how alg providers work/should work/be maintained?

This may ease and accelerate to start a discussion in the communities of
algs providers like OTB, GRASS, SAGA and others how to proceed in this
inter-project challenge.

If not, this will mean effectively missing
hundreds of algs from QGIS.

this is part of QGIS success and this question has also to be answered by
the QGIS community.

This may be an indication that OSGeo's inter-project communication should be
proactively improved in the future.....

-----
best regards
Helmut
--
Sent from: http://osgeo-org.1560.x6.nabble.com/Grass-Dev-f3991897.html

3

Hi,

2018-02-06 23:12 GMT+01:00 Helmut Kudrnovsky <hellik@web.de>:

this is part of QGIS success and this question has also to be answered by
the QGIS community.

This may be an indication that OSGeo's inter-project communication should be
proactively improved in the future.....

could be probably nice GSoC project too (GRASS provider in QGIS3). Ma

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

4

Martin Landa wrote

Hi,

2018-02-06 23:12 GMT+01:00 Helmut Kudrnovsky &lt;

hellik@

&gt;:

this is part of QGIS success and this question has also to be answered by
the QGIS community.

This may be an indication that OSGeo's inter-project communication should
be
proactively improved in the future.....

could be probably nice GSoC project too (GRASS provider in QGIS3). Ma

yes, that is part of the 'proactively thinking' what I mean.

not all possible opportunities to improve things are considered
beforehand...

-----
best regards
Helmut
--
Sent from: http://osgeo-org.1560.x6.nabble.com/Grass-Dev-f3991897.html

5

2018-02-06 23:34 GMT+01:00 Helmut Kudrnovsky <hellik@web.de>:

yes, that is part of the 'proactively thinking' what I mean.

draft of idea added [1]. Feel free to improve/extend :slight_smile:

Ma

[1] https://trac.osgeo.org/grass/wiki/GSoC/2018#ImproveGRASSintegrationinQGIS3

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

6

On Tue, Feb 6, 2018 at 11:46 PM, Martin Landa <landa.martin@gmail.com>
wrote:

2018-02-06 23:34 GMT+01:00 Helmut Kudrnovsky <hellik@web.de>:
> yes, that is part of the 'proactively thinking' what I mean.

draft of idea added [1]. Feel free to improve/extend :slight_smile:

I support this idea for GSoC

Ma

[1] https://trac.osgeo.org/grass/wiki/GSoC/2018#
ImproveGRASSintegrationinQGIS3

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

--
Regards,
   Rashad

7

Hello victor,

Do you see a possibility of a generic processing provider?. One that can read descriptor files and run algorithms!.
I see processing as a single plugin (included in QGIS or not). And if it can avoid need to have sub-plugin managed by all those other development team. That’s even better!.
Whichever toolbox want to be integrated into processing plugin can provide and maintain descriptor files individually. No QGIS developers need to be involved.
This way, external toolboxes’ team or qgis does not need to maintain/fix qgis--provider-plugin.

search → download → install plugin will be changed to configure providers install location.
If needed QGIS can control list of available providers (just names).

External tools’ dev team needs to know something such as spec of descriptor files, how to mention name of executable etc.

They don’t need to know stuff like how qgis run a processing algorithm, and things working of modeler, running with other algorithms.

Anyway, this is just an idea and don’t know what will be technically challenging issues?

The other way of putting processing plugin into core and providers classified as external plugins can avoid maintenance on qgis.
But this “strategy” can result in dead code and users complaining on both projects. Because, at some developers cannot manage project release + a plugin for qgis, another plugin for matlab or whatever
Putting all stuff in qgis tree and taking no responsibility of providers isn’t good either.

This way, each team takes part and result in better collaboration and contribution.

As time goes, generic descriptor provider gets better and stronger. Toolboxes are free to add, remove, modify their applications and users from both community benefit best of both worlds.

···

On Wed, Feb 7, 2018 at 11:18 AM, Victor Olaya <volayaf@gmail.com> wrote:

I dont see the advantage in having providers in core. And if there is an advantage, it’s clearly not in how easy it is going to be to maintain the plugin. If the people responsible of a given backend (like OTB) are going to maintain it (which makes sense), why putting it in core where they don’t have write access? Better in a separate repo. Also, they can release whenever there are changes, without having to wait for a new release. That way, the plugin will always be in sync with new releases of the backend app.

If we put them in core…why putting only this big ones (which in some cases require installing external apps manually by the user), and not put other plugins that exist and contain Processing providers?

I thought the idea was to not have core plugins and avoid them if possible. I don’t see why we have to treat plugins that have Processing provider in a different way. Specially considering how easy it is to install a plugin in QGIS.

Cheers

–

2018-02-06 18:57 GMT+01:00 Paolo Cavallini <cavallini@faunalia.it>:

Hi all,
following the discussion on qgis-dev ML:
https://lists.osgeo.org/pipermail/qgis-developer/2018-January/051701.html
it has been proposed to remove all external providers (namely OTB,
already removed, GRASS, and SAGA) from Processing in QGIS2 (GDAL will
remain as QGIS cannot be built without). This raised a number of
concerns, so PSC decided not to rush removing them from the upcoming 3.0
release, and to open a wider discussions about this, involving all the
interested parties, to find an optimal solution.
If volunteers outside QGIS core team, ideally from the respective
backends, could take the maintenance of these providers, not much would
change for the end user. If not, this will mean effectively missing
hundreds of algs from QGIS.
I personally propose to reinstate the OTB plugin with Rashad (from OTB)
as maintainer.
QGIS.ORG will seek to support any decision made with funding where possible.
Looking forward for your feedback.
All the best.

Paolo Cavallini - www.faunalia.eu
QGIS & PostGIS courses: http://www.faunalia.eu/training.html
https://www.google.com/trends/explore?date=all&geo=IT&q=qgis,arcgis

Regards,
Rashad

8

On Wed, Feb 7, 2018 at 3:03 PM, Victor Olaya <volayaf@gmail.com> wrote:

I dont think that it is possible to make it more generic. It's not only
descriptors with only outputs and inputs. Each backend app has its own
logic. GRASS needs outputs to be converted to its own format. SAGA accepts
only SHP for vector layers and generates its own SDAT format for raster
outputs. Parameters are also not passed in the same way to all apps. SAGA
has extent parameters splitted in 4 params with bbox coordinates. Each
provider has a different way of indicating a boolean value in the console
call. In short, the logic must be there somehow, specific for the provider,

can you confirm that a GRASS input/output parameter can solve their issue?.
Still there is SAGA and other stuff. So generic provider is not something
QGIS team can do. But I would like to know about this on GRASS issue.

What is the difference between maintaining a set of descriptor files (as
you propose) and a set of descriptor files + a class extending
GeoAlgorithmProvider with the logic (as it happens now)?. I think it is
easier to have a solid provider class for OTB and then do not ever change
it (assuming OTB syntax will never change), than having a generic provider,
which looks rather unfeasible.

Still OTB requires some changes in processing plugin to work. the old way

of twisting application only for QGIS must be avoided.
Without such a change there is no long-term existence even as plugin. Or
worse, it exists and will be broken.
QGIS must recognize such a behavior and avoid adding burden on external
toolboxes' developer teams by asking for splitting applications. Be it OTB,
GRASS, SAGA whatever.

While I was at it, I saw there is less stuff to be managed and can be done
using a customwidgetwrapper for OTB. because a custom widget wrapper works
in a special way only for one provider.
Hence the idea of generic provider came up!. In case of GRASS its
input/output parameter, for OTB it is selection list parameter.

Thanks for your valuable feedback on this. I am sure an idea of generic
provider came up sometime during your work on processing plugin.
It tough and need more expert work and safe is to avoid it totally. I agree
on that part.

As you say, there is the risk for dead code. In that case, i think it is
much better to have the provider outside of QGIS core, as a plugin. There
are dozens of dead plugins, and that is not nice, but it's kinda
acceptable. Having dead code in QGIS it's a much bigger issue, and we must
avoid that.

Cheers

2018-02-07 14:41 GMT+01:00 Rashad Kanavath <mohammedrashadkm@gmail.com>:

Hello victor,

Do you see a possibility of a generic processing provider?. One that can
read descriptor files and run algorithms!.
I see processing as a single plugin (included in QGIS or not). And if it
can avoid need to have sub-plugin managed by all those other development
team. That's even better!.
Whichever toolbox want to be integrated into processing plugin can
provide and maintain descriptor files individually. No QGIS developers need
to be involved.
This way, external toolboxes' team or qgis does not need to maintain/fix
qgis-<TOOLBOX>-provider-plugin.

search -> download -> install plugin will be changed to configure
providers install location.
If needed QGIS can control list of available providers (just names).

External tools' dev team needs to know something such as spec of
descriptor files, how to mention name of executable etc.
They don't need to know stuff like how qgis run a processing algorithm,
and things working of modeler, running with other algorithms.

Anyway, this is just an idea and don't know what will be technically
challenging issues?

The other way of putting processing plugin into core and providers
classified as external plugins can avoid maintenance on qgis.
But this "strategy" can result in dead code and users complaining on both
projects. Because, at some developers cannot manage project release + a
plugin for qgis, another plugin for matlab or whatever
Putting all stuff in qgis tree and taking no responsibility of providers
isn't good either.

This way, each team takes part and result in better collaboration and
contribution.
As time goes, generic descriptor provider gets better and stronger.
Toolboxes are free to add, remove, modify their applications and users from
both community benefit best of both worlds.

On Wed, Feb 7, 2018 at 11:18 AM, Victor Olaya <volayaf@gmail.com> wrote:

I dont see the advantage in having providers in core. And if there is an
advantage, it's clearly not in how easy it is going to be to maintain the
plugin. If the people responsible of a given backend (like OTB) are going
to maintain it (which makes sense), why putting it in core where they don't
have write access? Better in a separate repo. Also, they can release
whenever there are changes, without having to wait for a new release. That
way, the plugin will always be in sync with new releases of the backend app.

If we put them in core...why putting only this big ones (which in some
cases require installing external apps manually by the user), and not put
other plugins that exist and contain Processing providers?

I thought the idea was to not have core plugins and avoid them if
possible. I don't see why we have to treat plugins that have Processing
provider in a different way. Specially considering how easy it is to
install a plugin in QGIS.

Cheers

2018-02-06 18:57 GMT+01:00 Paolo Cavallini <cavallini@faunalia.it>:

Hi all,
following the discussion on qgis-dev ML:
https://lists.osgeo.org/pipermail/qgis-developer/2018-Januar
y/051701.html
it has been proposed to remove all external providers (namely OTB,
already removed, GRASS, and SAGA) from Processing in QGIS2 (GDAL will
remain as QGIS cannot be built without). This raised a number of
concerns, so PSC decided not to rush removing them from the upcoming 3.0
release, and to open a wider discussions about this, involving all the
interested parties, to find an optimal solution.
If volunteers outside QGIS core team, ideally from the respective
backends, could take the maintenance of these providers, not much would
change for the end user. If not, this will mean effectively missing
hundreds of algs from QGIS.
I personally propose to reinstate the OTB plugin with Rashad (from OTB)
as maintainer.
QGIS.ORG will seek to support any decision made with funding where
possible.
Looking forward for your feedback.
All the best.
--
Paolo Cavallini - www.faunalia.eu
QGIS & PostGIS courses: http://www.faunalia.eu/training.html
https://www.google.com/trends/explore?date=all&geo=IT&q=qgis,arcgis

--
Regards,
   Rashad

--
Regards,
   Rashad

9

Il 07/02/2018 11:18, Victor Olaya ha scritto:

I dont see the advantage in having providers in core.

I see the following:
* tests (already available in our infrastructure)
* translations
* more exposure
* documentation

And if there is an
advantage, it's clearly not in how easy it is going to be to maintain
the plugin.

until now it has been maintained somehow; if more resources are needed,
we can find a way

If the people responsible of a given backend (like OTB) are
going to maintain it (which makes sense), why putting it in core where
they don't have write access?

why not granting them write access?

Better in a separate repo. Also, they can
release whenever there are changes, without having to wait for a new
release. That way, the plugin will always be in sync with new releases
of the backend app.

this is certainly true; AFAICT OTB people has proposed a solution

If we put them in core...why putting only this big ones (which in some
cases require installing external apps manually by the user), and not
put other plugins that exist and contain Processing providers?

I'd be in favour of adding anything important for users.

Thanks for your thoughts.

When in Madeira we can have a discussion about this. It would be good if
all interested parties could meet, locally and remotely.

All the best.
--
Paolo Cavallini - www.faunalia.eu
QGIS & PostGIS courses: http://www.faunalia.eu/training.html
https://www.google.com/trends/explore?date=all&geo=IT&q=qgis,arcgis

10

I’m much more in favour for out of core providers, for the same reasons reported by Victor. Having only GDAL and QGIS algorithms in core will reduce the number of available algorithms out of the box, but:

  • from my experience - comprising years of feedbacks from the courses I teach - the most frequently used algs are available within the GDAL and QGIS algs list

  • a few generic and widely used algs are from GRASS and SAGA. We would miss them - out of the box - but it could also be an opportunity to port them. For examples v.to.points, transects, profiles.
    Anyway we would have the plugins for GRASS and SAGA, in case…

  • specific algs are for specialized users. Here I think plugins are best suited (OTB, R, TauDEM, etc.). Tomorrow a new sw could be added to the list, consistently with the others approach.

I appreciate a lot the work from Richard, I hope this discussion is not understood as to belittle its effort!

my 2 cents…
Giovanni

···

Il 7 feb 2018 18:25, “Paolo Cavallini” <cavallini@faunalia.it> ha scritto:

Il 07/02/2018 11:18, Victor Olaya ha scritto:

I dont see the advantage in having providers in core.

I see the following:

  • tests (already available in our infrastructure)
  • translations
  • more exposure
  • documentation

And if there is an
advantage, it’s clearly not in how easy it is going to be to maintain
the plugin.

until now it has been maintained somehow; if more resources are needed,
we can find a way

If the people responsible of a given backend (like OTB) are
going to maintain it (which makes sense), why putting it in core where
they don’t have write access?

why not granting them write access?

Better in a separate repo. Also, they can
release whenever there are changes, without having to wait for a new
release. That way, the plugin will always be in sync with new releases
of the backend app.

this is certainly true; AFAICT OTB people has proposed a solution

If we put them in core…why putting only this big ones (which in some
cases require installing external apps manually by the user), and not
put other plugins that exist and contain Processing providers?

I’d be in favour of adding anything important for users.

Thanks for your thoughts.

When in Madeira we can have a discussion about this. It would be good if
all interested parties could meet, locally and remotely.

All the best.

Paolo Cavallini - www.faunalia.eu
QGIS & PostGIS courses: http://www.faunalia.eu/training.html
https://www.google.com/trends/explore?date=all&geo=IT&q=qgis,arcgis

QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

11

Hi all,
I disagree heartily: without GRASS and SAGA QGIS is currently unsuitable
for serious, comprehensive GIS analysis work. Full stop.
Missing one specific alg, even if not used daily, means having to switch
to other software.
We have already removed R provider: even if used by a tiny minority, and
certainly not perfect, the previous situation was better than the
current one, without the option of using R from QGIS.
I think we have to be extra cautious on this ground.
All the best.

Il 07/02/2018 19:07, G. Allegri ha scritto:

- from my experience - comprising years of feedbacks from the courses I
teach - the most frequently used algs are available within the GDAL and
QGIS algs list

- a few generic and widely used algs are from GRASS and SAGA. We would
miss them - out of the box - but it could also be an opportunity to port
them. For examples v.to.points, transects, profiles.
Anyway we would have the plugins for GRASS and SAGA, in case...

- specific algs are for specialized users. Here I think plugins are best
suited (OTB, R, TauDEM, etc.). Tomorrow a new sw could be added to the
list, consistently with the others approach.

I appreciate a lot the work from Richard, I hope this discussion is not
understood as to belittle its effort!

--
Paolo Cavallini - www.faunalia.eu
QGIS & PostGIS courses: http://www.faunalia.eu/training.html
https://www.google.com/trends/explore?date=all&geo=IT&q=qgis,arcgis

12

On Wed, Feb 7, 2018 at 7:07 PM, G. Allegri <giohappy@gmail.com> wrote:

I'm much more in favour for out of core providers, for the same reasons
reported by Victor. Having only GDAL and QGIS algorithms in core will
reduce the number of available algorithms out of the box, but:

- from my experience - comprising years of feedbacks from the courses I
teach - the most frequently used algs are available within the GDAL and
QGIS algs list

Do you have these toolboxes installed during training? OTB, SAGA, GRASS
etc..
OTB is focused on remote sensing. Unless you have a training or course that
area, your statistics on them being not needed is pretty unreliable. Don't
you think?
What QGIS uses to run a classification/segmentation algorithm without OTB
and GRASS GIS.

- a few generic and widely used algs are from GRASS and SAGA. We would
miss them - out of the box - but it could also be an opportunity to port
them. For examples v.to.points, transects, profiles.
Anyway we would have the plugins for GRASS and SAGA, in case...

- specific algs are for specialized users. Here I think plugins are best
suited (OTB, R, TauDEM, etc.). Tomorrow a new sw could be added to the
list, consistently with the others approach.

I appreciate a lot the work from Richard, I hope this discussion is not
understood as to belittle its effort!

my 2 cents...
Giovanni

Il 7 feb 2018 18:25, "Paolo Cavallini" <cavallini@faunalia.it> ha scritto:

Il 07/02/2018 11:18, Victor Olaya ha scritto:
> I dont see the advantage in having providers in core.

I see the following:
* tests (already available in our infrastructure)
* translations
* more exposure
* documentation

> And if there is an
> advantage, it's clearly not in how easy it is going to be to maintain
> the plugin.

until now it has been maintained somehow; if more resources are needed,
we can find a way

> If the people responsible of a given backend (like OTB) are
> going to maintain it (which makes sense), why putting it in core where
> they don't have write access?

why not granting them write access?

> Better in a separate repo. Also, they can
> release whenever there are changes, without having to wait for a new
> release. That way, the plugin will always be in sync with new releases
> of the backend app.

this is certainly true; AFAICT OTB people has proposed a solution

> If we put them in core...why putting only this big ones (which in some
> cases require installing external apps manually by the user), and not
> put other plugins that exist and contain Processing providers?

I'd be in favour of adding anything important for users.

Thanks for your thoughts.

When in Madeira we can have a discussion about this. It would be good if
all interested parties could meet, locally and remotely.

All the best.
--
Paolo Cavallini - www.faunalia.eu
QGIS & PostGIS courses: http://www.faunalia.eu/training.html
https://www.google.com/trends/explore?date=all&geo=IT&q=qgis,arcgis
_______________________________________________
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

--
Regards,
   Rashad

13

Hi,

Regarding the negative effect of algorithms getting lost I fully agree with you Paolo.

However, it might simplify the discussion if we try separate user experience and development (and packaging) solutions as well as means and ends...

Aim should be to have the different back-ends available for user in a way that is as straight forward as possible. From my point of view that includes that possibly available providers are not hidden from users who just install bare QGIS. If they want to activate them, but don`t have the backends installed (and possibly a plugin) then they should get a notice that they have to install them (and I don`t see a problem with installing provider + plugin compared to just provider).

And if that sort of user experience is guaranteed I - as a user - would not care about where the code is maintained (in QGIS core, in the provider core, or in a separate plugin). My impression is that Victors arguments for an out-of-core solution are very valid, esp. given effects of the independent release cycles of the backends and QGIS on packaging needs (at least for the GRASS plugin).
The only difference I see is that additional packages would be needed for a plugin solution. But that is probably not too bad or even an advantage...

Finally, if there is interest in keeping the processing integration alive (and it obviously is) having it in an independent repo should not be a show stopper. Only negative effect I can see is that this produces additional repos, where access rights have to be managed. But that should not be a major issue either...

Cheers
Stefan

P.S.: Just a pity that this discussion starts after medspx just put down all this work:
https://github.com/qgis/QGIS/pull/5603
https://github.com/qgis/QGIS/pull/5968
https://github.com/qgis/QGIS/pull/5426

-----Original Message-----
From: grass-dev [mailto:grass-dev-bounces@lists.osgeo.org] On Behalf Of Paolo Cavallini
Sent: torsdag 8. februar 2018 09.04
To: G. Allegri <giohappy@gmail.com>
Cc: qgis-developer <qgis-developer@lists.osgeo.org>; saga-gis-developer@lists.sourceforge.net; grass-dev <grass-dev@lists.osgeo.org>; Victor Olaya <volayaf@gmail.com>
Subject: Re: [GRASS-dev] [QGIS-Developer] External providers in QGIS

Hi all,
I disagree heartily: without GRASS and SAGA QGIS is currently unsuitable for serious, comprehensive GIS analysis work. Full stop.
Missing one specific alg, even if not used daily, means having to switch to other software.
We have already removed R provider: even if used by a tiny minority, and certainly not perfect, the previous situation was better than the current one, without the option of using R from QGIS.
I think we have to be extra cautious on this ground.
All the best.

Il 07/02/2018 19:07, G. Allegri ha scritto:

- from my experience - comprising years of feedbacks from the courses
I teach - the most frequently used algs are available within the GDAL
and QGIS algs list

- a few generic and widely used algs are from GRASS and SAGA. We would
miss them - out of the box - but it could also be an opportunity to
port them. For examples v.to.points, transects, profiles.
Anyway we would have the plugins for GRASS and SAGA, in case...

- specific algs are for specialized users. Here I think plugins are
best suited (OTB, R, TauDEM, etc.). Tomorrow a new sw could be added
to the list, consistently with the others approach.

I appreciate a lot the work from Richard, I hope this discussion is
not understood as to belittle its effort!

--
Paolo Cavallini - www.faunalia.eu
QGIS & PostGIS courses: http://www.faunalia.eu/training.html
https://www.google.com/trends/explore?date=all&geo=IT&q=qgis,arcgis
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

14

On Wed, Feb 7, 2018 at 6:25 PM, Paolo Cavallini <cavallini@faunalia.it>
wrote:

Il 07/02/2018 11:18, Victor Olaya ha scritto:
> I dont see the advantage in having providers in core.

I see the following:
* tests (already available in our infrastructure)
* translations
* more exposure
* documentation

> And if there is an
> advantage, it's clearly not in how easy it is going to be to maintain
> the plugin.

until now it has been maintained somehow; if more resources are needed,
we can find a way

> If the people responsible of a given backend (like OTB) are
> going to maintain it (which makes sense), why putting it in core where
> they don't have write access?

why not granting them write access?

That would still need users *waiting* for QGIS release for fix in algo is
what I understood from other parts of discussion.
I don't know what these developers are going to do with a bugfix after a
new release. That's some kind of mystery unsolved to me.
I hope there will be zero bugs after releases.

> Better in a separate repo. Also, they can
> release whenever there are changes, without having to wait for a new
> release. That way, the plugin will always be in sync with new releases
> of the backend app.

this is certainly true; AFAICT OTB people has proposed a solution

> If we put them in core...why putting only this big ones (which in some
> cases require installing external apps manually by the user), and not
> put other plugins that exist and contain Processing providers?

I'd be in favour of adding anything important for users.

Thanks for your thoughts.

When in Madeira we can have a discussion about this. It would be good if
all interested parties could meet, locally and remotely.

All the best.
--
Paolo Cavallini - www.faunalia.eu
QGIS & PostGIS courses: http://www.faunalia.eu/training.html
https://www.google.com/trends/explore?date=all&geo=IT&q=qgis,arcgis

--
Regards,
   Rashad

15

What I ment is that, from my perspective, we shouldn’t seek to make QGIS a do-it-all software by default, because the GIS/EO/RS/Data Analysis fields are so wide that, from that point of view, everything could go into QGIS and it would be an overwhelmin experience for users.
Any generic sw I know offers extensions, plugins, for specialized use cases. From my experience a user prefers few tools that match their needs, instead of digging into a long list of (mostly) obscure algs…

Anyway, I don’t want to stress on this…
Giovanni

···

2018-02-08 10:15 GMT+01:00 Stefan Blumentrath <Stefan.Blumentrath@nina.no>:

Hi,

Regarding the negative effect of algorithms getting lost I fully agree with you Paolo.

However, it might simplify the discussion if we try separate user experience and development (and packaging) solutions as well as means and ends…

Aim should be to have the different back-ends available for user in a way that is as straight forward as possible. From my point of view that includes that possibly available providers are not hidden from users who just install bare QGIS. If they want to activate them, but dont have the backends installed (and possibly a plugin) then they should get a notice that they have to install them (and I dont see a problem with installing provider + plugin compared to just provider).

And if that sort of user experience is guaranteed I - as a user - would not care about where the code is maintained (in QGIS core, in the provider core, or in a separate plugin). My impression is that Victors arguments for an out-of-core solution are very valid, esp. given effects of the independent release cycles of the backends and QGIS on packaging needs (at least for the GRASS plugin).
The only difference I see is that additional packages would be needed for a plugin solution. But that is probably not too bad or even an advantage…

Finally, if there is interest in keeping the processing integration alive (and it obviously is) having it in an independent repo should not be a show stopper. Only negative effect I can see is that this produces additional repos, where access rights have to be managed. But that should not be a major issue either…

Cheers
Stefan

P.S.: Just a pity that this discussion starts after medspx just put down all this work:
https://github.com/qgis/QGIS/pull/5603
https://github.com/qgis/QGIS/pull/5968
https://github.com/qgis/QGIS/pull/5426

-----Original Message-----
From: grass-dev [mailto:grass-dev-bounces@lists.osgeo.org] On Behalf Of Paolo Cavallini
Sent: torsdag 8. februar 2018 09.04
To: G. Allegri <giohappy@gmail.com>
Cc: qgis-developer <qgis-developer@lists.osgeo.org>; saga-gis-developer@lists.sourceforge.net; grass-dev <grass-dev@lists.osgeo.org>; Victor Olaya <volayaf@gmail.com>
Subject: Re: [GRASS-dev] [QGIS-Developer] External providers in QGIS

Hi all,
I disagree heartily: without GRASS and SAGA QGIS is currently unsuitable for serious, comprehensive GIS analysis work. Full stop.
Missing one specific alg, even if not used daily, means having to switch to other software.
We have already removed R provider: even if used by a tiny minority, and certainly not perfect, the previous situation was better than the current one, without the option of using R from QGIS.
I think we have to be extra cautious on this ground.
All the best.

Il 07/02/2018 19:07, G. Allegri ha scritto:

  • from my experience - comprising years of feedbacks from the courses
    I teach - the most frequently used algs are available within the GDAL
    and QGIS algs list

  • a few generic and widely used algs are from GRASS and SAGA. We would
    miss them - out of the box - but it could also be an opportunity to
    port them. For examples v.to.points, transects, profiles.
    Anyway we would have the plugins for GRASS and SAGA, in case…

  • specific algs are for specialized users. Here I think plugins are
    best suited (OTB, R, TauDEM, etc.). Tomorrow a new sw could be added
    to the list, consistently with the others approach.

I appreciate a lot the work from Richard, I hope this discussion is
not understood as to belittle its effort!

–
Paolo Cavallini - www.faunalia.eu
QGIS & PostGIS courses: http://www.faunalia.eu/training.html
https://www.google.com/trends/explore?date=all&geo=IT&q=qgis,arcgis

grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

16

On Thu, Feb 8, 2018 at 10:15 AM, Stefan Blumentrath <
Stefan.Blumentrath@nina.no> wrote:

Hi,

Regarding the negative effect of algorithms getting lost I fully agree
with you Paolo.

However, it might simplify the discussion if we try separate user
experience and development (and packaging) solutions as well as means and
ends...

Aim should be to have the different back-ends available for user in a way
that is as straight forward as possible. From my point of view that
includes that possibly available providers are not hidden from users who
just install bare QGIS. If they want to activate them, but don`t have the
backends installed (and possibly a plugin) then they should get a notice
that they have to install them (and I don`t see a problem with installing
provider + plugin compared to just provider).

OTB's proposed solution was that plugin or provider algorithm if activated
can find otb installation. If not found, there is code which will download
and install otb packages and configure it for users.

And if that sort of user experience is guaranteed I - as a user - would
not care about where the code is maintained (in QGIS core, in the provider
core, or in a separate plugin). My impression is that Victors arguments for
an out-of-core solution are very valid, esp. given effects of the
independent release cycles of the backends and QGIS on packaging needs (at
least for the GRASS plugin).
The only difference I see is that additional packages would be needed for
a plugin solution. But that is probably not too bad or even an advantage...

Finally, if there is interest in keeping the processing integration alive
(and it obviously is) having it in an independent repo should not be a show
stopper. Only negative effect I can see is that this produces additional
repos, where access rights have to be managed. But that should not be a
major issue either...

Cheers
Stefan

P.S.: Just a pity that this discussion starts after medspx just put down
all this work:
https://github.com/qgis/QGIS/pull/5603
https://github.com/qgis/QGIS/pull/5968
https://github.com/qgis/QGIS/pull/5426

-----Original Message-----
From: grass-dev [mailto:grass-dev-bounces@lists.osgeo.org] On Behalf Of
Paolo Cavallini
Sent: torsdag 8. februar 2018 09.04
To: G. Allegri <giohappy@gmail.com>
Cc: qgis-developer <qgis-developer@lists.osgeo.org>;
saga-gis-developer@lists.sourceforge.net; grass-dev <
grass-dev@lists.osgeo.org>; Victor Olaya <volayaf@gmail.com>
Subject: Re: [GRASS-dev] [QGIS-Developer] External providers in QGIS

Hi all,
I disagree heartily: without GRASS and SAGA QGIS is currently unsuitable
for serious, comprehensive GIS analysis work. Full stop.
Missing one specific alg, even if not used daily, means having to switch
to other software.
We have already removed R provider: even if used by a tiny minority, and
certainly not perfect, the previous situation was better than the current
one, without the option of using R from QGIS.
I think we have to be extra cautious on this ground.
All the best.

Il 07/02/2018 19:07, G. Allegri ha scritto:

> - from my experience - comprising years of feedbacks from the courses
> I teach - the most frequently used algs are available within the GDAL
> and QGIS algs list
>
> - a few generic and widely used algs are from GRASS and SAGA. We would
> miss them - out of the box - but it could also be an opportunity to
> port them. For examples v.to.points, transects, profiles.
> Anyway we would have the plugins for GRASS and SAGA, in case...
>
> - specific algs are for specialized users. Here I think plugins are
> best suited (OTB, R, TauDEM, etc.). Tomorrow a new sw could be added
> to the list, consistently with the others approach.
>
> I appreciate a lot the work from Richard, I hope this discussion is
> not understood as to belittle its effort!

--
Paolo Cavallini - www.faunalia.eu
QGIS & PostGIS courses: http://www.faunalia.eu/training.html
https://www.google.com/trends/explore?date=all&geo=IT&q=qgis,arcgis
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

--
Regards,
   Rashad

17

On Thu, Feb 8, 2018 at 11:32 AM, Victor Olaya <volayaf@gmail.com> wrote:

OTB's proposed solution was that plugin or provider algorithm if
activated can find otb installation. If not found, there is code which will
download and install otb packages and configure it for users.

I still don't see why this cannot be done if OTB provider is a separate
plugin. Users can install a plugin with the provider, and then that
provider can have the logic to automatically download OTB, install it, or
do whatever is needed in case OTB is not found already installed.

I think is is good to educate our users a little bit. We are talking about
people that will be using complex algorithms for performing advanced
analysis. I guess we can expect that they can install a plugin themselves
and it is not a traumatic experience for them... Looks like installing a
separate plugin it's some sort of cruel chinese torture...when it takes
just 3 mouse clicks.

I agree with Giovanni, there is no need to provide something that has
everything installed out-of the box. Also, take into account that reading
the files that contain the algorithm descriptions takes time (plus, there
might be some additional checks performed when populating the toolbox).
People not doing analysis should not have to suffer that extra loading
time...

QGIS increases no of algorithms for a provider. It is simple as that. OTB
has less than 80 applications and coming to QGIS, it will be 100 or 150.
(no interest to count them in qgis)
And OTB was reading descriptor from xml which is going to be now csv like
others.
Given all that info, I won't be surprised if it takes more time in future
because of new algorithms added to providers. If QGIS was reading proper
way or even open to accepting fixes.

The entire proposal/request to put back OTB was that decision was made on
OLD code. when the update code is there, it has less problems.
Based on concerns raised by other developers no matter if you can fix it,
they can't be put back. This single point was fueling this and other
threads.
Also discussion wasn't started with "why no providers are included?". It
was why some of them are removed even if there is an offer to help!

I don't care enough to continue this discussion about plugin or not plugin.

But aside all decision making stuff, can you check what is to be done in
this PR?
https://github.com/qgis/QGIS/pull/6272
It is something worthy a discussion and not a plugin or not. I was asking
because QGIS 3 is near and diff is not that big.
If there is something extra need to be done to get this merged, I would
happy to hear about it.

About the R plugin not being available now...well, it's in a separate repo
and can be adapted to QGIS 3, packaged and published. No one has taken care
of that, it's true, but that only means we have no R support. If the R
provider was be in core, it would mean that we would have a dead or
malfunctioning provider being shipped with QGIS. That is a much worse
option. And I dont see why, if someone is willing to fix that R provider,
having it in core will make it easier.

Cheers!

--
Regards,
   Rashad

18

On 08/02/18 13:43, Rashad Kanavath wrote:

On Thu, Feb 8, 2018 at 11:32 AM, Victor Olaya <volayaf@gmail.com <mailto:volayaf@gmail.com>> wrote:

        OTB's proposed solution was that plugin or provider algorithm if
        activated can find otb installation. If not found, there is code
        which will download and install otb packages and configure it
        for users.

    I still don't see why this cannot be done if OTB provider is a
    separate plugin. Users can install a plugin with the provider, and
    then that provider can have the logic to automatically download OTB,
    install it, or do whatever is needed in case OTB is not found
    already installed.

    I think is is good to educate our users a little bit. We are talking
    about people that will be using complex algorithms for performing
    advanced analysis. I guess we can expect that they can install a
    plugin themselves and it is not a traumatic experience for them...
    Looks like installing a separate plugin it's some sort of cruel
    chinese torture...when it takes just 3 mouse clicks.

    I agree with Giovanni, there is no need to provide something that
    has everything installed out-of the box. Also, take into account
    that reading the files that contain the algorithm descriptions takes
    time (plus, there might be some additional checks performed when
    populating the toolbox). People not doing analysis should not have
    to suffer that extra loading time...

QGIS increases no of algorithms for a provider. It is simple as that. OTB has less than 80 applications and coming to QGIS, it will be 100 or 150. (no interest to count them in qgis)
And OTB was reading descriptor from xml which is going to be now csv like others.
Given all that info, I won't be surprised if it takes more time in future because of new algorithms added to providers. If QGIS was reading proper way or even open to accepting fixes.

The entire proposal/request to put back OTB was that decision was made on OLD code. when the update code is there, it has less problems.
Based on concerns raised by other developers no matter if you can fix it, they can't be put back. This single point was fueling this and other threads.
Also discussion wasn't started with "why no providers are included?". It was why some of them are removed even if there is an offer to help!

I don't care enough to continue this discussion about plugin or not plugin.

Warning: I don't claim any knowledge of the actual QGIS provider code, but I'm afraid that this is the case for many external SW developers, so please bear with me and correct me where I'm wrong.

However, I do have the feeling that part of the debate is due to fuzzyness of the actual subject. AFAIU, there a different issues here, and only if we clarify what we are talking about exactly, and what the actual issues are, can we advance. So, here's my understanding:

1) the descriptor files of the different algorithms in external SW

It seems that at least from OTB and GRASS side, willingness has been signaled to take over the handling of those.

2) the code that allows to launch these algorithms from within QGIS

IIUC this is again subdivided into two parts:

- a generic class within QGIS code for creating a provider
- the specific external SW related instances of these classes

The first part is definitely part of QGIS core.

The current debate (again AFAIU) is mostly about the second part. And one question linked to this is how much of the handling can be done by the generic class code, and how much is SW-specific. IOW, would it be possible to have exactly the same format of descriptor files for all SW, and a generic API to interpret those, or are the idiosyncrasies of the different SW such that API and descriptor files will have to be SW specific ?

A second question is how much within the second part (the SW specific provider) depends on evolutions of QGIS code, i.e. when QGIS moves from one version to the next, will the code in this part have to change, or will it remain stable, and how much depends on evolutions in the external SW code. If changes in external SW happen more often than it seems to make sense to keep this part of the code away from QGIS core, but if changes in QGIS happen more often than the opposite would seem better.

Another issue is that current expertise on how to code these providers is within the QGIS development team. If you decide to put these providers into plugins, _and_ to no longer ensure the maintenance of these plugins, this would mean that the development teams of the external SW have to acquire the necessary knowledge. Looking at general scarcity of human power in all of the teams, I'm not sure this will happen easily.

Do I understand things correctly ?

Moritz

19

Dear all,

a case somewhat relevant to the discussion.

I am tasked by a client to work with and on some existing scripts that derive
maps related to recreational areas. These scripts use GRASS GIS.
Hence, it is naturally to shape a GRASS GIS Add-on.

The add-on will eventually be of interest for many, and mostly non
GIS-experts. The whole idea was to expose the Add-on to QGIS and
eventually train and practice with it persons who are not experienced
GIS users.

Both QGIS and GRASS GIS, in this case, and the whole OSGeo ecosystem,
have things to gain, while offering small tools that interest many.

I guess, there are numerous cases like this one. What would,
effectively, mean a removal of external providers (in this case GRASS
GIS)?

Among the, indeed, many algorithms that might appear, or be
overwhelming, there are always here and there tools that make life
easier for people who don't have the time or the need to dive deep into
special concepts of GRASS, SAGA or OTB. And QGIS is for the the
reference.

Thanks, Nikos

* Paolo Cavallini <cavallini@faunalia.it> [2018-02-06 18:57:17 +0100]:

Hi all,
following the discussion on qgis-dev ML:
https://lists.osgeo.org/pipermail/qgis-developer/2018-January/051701.html
it has been proposed to remove all external providers (namely OTB,
already removed, GRASS, and SAGA) from Processing in QGIS2 (GDAL will
remain as QGIS cannot be built without). This raised a number of
concerns, so PSC decided not to rush removing them from the upcoming 3.0
release, and to open a wider discussions about this, involving all the
interested parties, to find an optimal solution.
If volunteers outside QGIS core team, ideally from the respective
backends, could take the maintenance of these providers, not much would
change for the end user. If not, this will mean effectively missing
hundreds of algs from QGIS.
I personally propose to reinstate the OTB plugin with Rashad (from OTB)
as maintainer.
QGIS.ORG will seek to support any decision made with funding where possible.
Looking forward for your feedback.
All the best.
--
Paolo Cavallini - www.faunalia.eu
QGIS & PostGIS courses: http://www.faunalia.eu/training.html
https://www.google.com/trends/explore?date=all&geo=IT&q=qgis,arcgis
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

--
Nikos Alexandris | Remote Sensing & Geomatics
GPG Key Fingerprint 6F9D4506F3CA28380974D31A9053534B693C4FB3

20

On 08/02/18 15:08, Nikos Alexandris wrote:

I guess, there are numerous cases like this one. What would,
effectively, mean a removal of external providers (in this case GRASS
GIS)?

Let's make it clear once and for all: noone is speaking about the complete removal of external providers. The issue is where, how and by whom they should be maintained, i.e. within the QGIS core code base, or as plugins.

Moritz