···
On Wed, Feb 7, 2018 at 12:32 AM, Vaclav Petras <wenzeslaus@gmail.com> wrote:
Dear all,
I’m answering here rather than the “[GRASS-dev] External providers in QGIS” thread since here we have the technical discussion already. I’m also removing grass-user list.
OTB also suffers from this issue in QGIS. We are solving it with new updates to processing provider. Maybe grass could reuse it!. Currently, I have two blocking issues.
- QGIS batch processing that demands a table with each row representing a single execution of algorithm.
- https://github.com/qgis/QGIS/pull/6272
standard processing dialog and qgis graphical modeler works after pull request (pending merge)
For OTB, my initial thought is to disable batch processing and later work on this part. It is important to have two of them working than none of them.
QGIS’s proposition earlier was to split application which I don’t think is something right to do here.
Since grass also has similar issues, I prefer QGIS would accept fixes.
If there is an executable or algorithm found in a gis provider, it can ask what should I do? And the provider can answer run some program with so and so argument(s).
In case of otb, there will be a dedicated executable to deal with. IIUC grass can say, if you have an algorithm in a directory (listed in provider, could be relative to install prefix of GRASS),
run <module_name> --qgis-descriptor. With this, a user installing an addon will have to do nothing to make it work with qgis.
ofcourse, once qgis found a descriptor file for an algorithm, it won’t bother ask provider what to do again or next run. Because it can read this file and run algorithm.
The above “template” of identifying a QGIS provider algorithm can be generic and reused in different providers.
If there is question of “trusting” on running apps in a given directory, that is something debatable.
–
Regards,
Rashad
On Mon, Feb 5, 2018 at 3:56 PM, Stefan Blumentrath <Stefan.Blumentrath@nina.no> wrote:
The text descriptor files have been a real pain for maintaining interfaces for GRASS integration in Processing as well as the GRASS plugin. In addition, they make usage of AddOns practically impossible for most of the QGIS users.
There were already suggestions to change it in the past to something based on “–interface-description”, but it always hit various issues like those things you mention below. However, I think that it is exactly what needs to be done in any case because only that is sustainable and potentially also reusable by other tools (besides QGIS). The reuse is quite interesting also in the relation to the automatic import, export and temporary location (in context of --exec or grass_session).
However, for QGIS (and this is true for both integrations), the module UI was deliberately simplified by hiding / removing “advanced” option or splitting modules into “sub-types”.
Here the question is how important it is to do (keep) this or if perhaps different solution would give better result. For example, splitting (in the QGIS interface) r.slope.aspect to r.slope and r.aspect (and more) makes it slower for the user to compute slope and aspect - 2x filling the form and 2x loading input data. Reorganization of r.slope.aspect interface may result in more convenient form and shorter computation than the “r.slope + r.aspect” solution.
In addition not all modules could be meaningful added to the two QGIS integrations (e.g. temporal modules in Processing).
I think there needs to be a white list or back list. It seems analogous to the “toolboxes” for wxGUI. All r3.* and t.* modules are out for QGIS, but I think the rest needs to be hand picked. The limitation of the hand picking is that it does not work for addons and custom modules in general.
And finally, some require extra work on the QGIS side (like r.mapcalc).
We should address this ASAP in the process. Besides r.mapcalc, it is esp. modules which output list of maps which are either time series (e.g. r.sim.water with -t) or multiple results (e.g. r.texture method=asm,var output=basename). (There is a discussion about this somewhere on the list.) I’m not sure what is the current solution for i.* which take a group, perhaps a list or a multiband file is good (?). In relation that, t.* and r3.* modules could take and output a list of maps (multiband) as well.
So, I would assume that a --qgis-descriptor solution would be most appropriate.
It can be “m.odule.name --qgis-descriptor” or “g.get.interface module=m.odule.name format=qgis” which can take the general (or generalized) --interface-description and transform it to what the processing adapter tool needs (or it can be just part of the adapter).
But that would still require additional work (beyond implementing a parser solution) if the principles for the GRASS module UI in QGIS should stay as it is, like:
- Tagging options and flags as advanced or basic/main/common
We already have the sections and also required parameters marked (in GUI), so the question is why it is not enough. Perhaps just fixing that is enough (for example description for each group). But of course if we say that the requirement for QGIS alg is that there is less than 5 options and no flags, then we need some additional system.
(could be an opportunity to consolidate terminology in the module UI in GRASS as well)
I think one of the challenges brought up in the past was GRASS terminology versus QGIS terminology. We don’t have a comparison table for that, but we have one for ArcGIS [1] and there are some obvious things like “vector map” versus “vector layer” and than GRASS’ “layer of vector map”.
I can see 3 solutions: 1) ignore the differences, 2) come up with different names (I think we have been there already), or 3) have a special description field which is QGIS-friendly or OGC glossary compliant if needed.
[1] https://grasswiki.osgeo.org/wiki/Terminology_comparison_between_ArcGIS_and_GRASS_GIS
- Deciding which modules to use / exclude from QGIS (and how to mark them)
We have the “toolbox” mechanism to populate things in wxGUI, so the files can be reused to create algorithm tree for QGIS. One needs to add the modules there explicitly and then additionally find addons, but we do that for wxGUI now.
Maybe also Ondrejs work could be useful here : https://trac.osgeo.org/grass/wiki/GSoC/2016/PyQtGUI ?
Yes, for the QGIS GRASS plugin. I’m not sure how much the plugin is part of this discussion. Not long ago, Radim worked on it a lot. Ondrej’s work is aimed at what the C++ plugin is aimed at and what wxGUI is aimed at - i.e. it’s a Qt interface to GRASS which could be used in GRASS or as QGIS plugin. I don’t think it is useful for the processing part, but it is a good prototype to start from. Currently it misses a lot of features, it is basically limited reimplementation of gui/…/forms.py. There is also some raster rendering work from Soeren which is parallel to the C++/C part of the QGIS GRASS plugin.
Best,
Vaclav
Cheers
Stefan
From: grass-dev [mailto:grass-dev-bounces@lists.osgeo.org] On Behalf Of Rashad Kanavath
Sent: mandag 5. februar 2018 17.06
To: Moritz Lennert <mlennert@club.worldonline.be>
Cc: grass-user@lists.osgeo.org; grass-dev@lists.osgeo.org; Helmut Kudrnovsky <hellik@web.de>
Subject: Re: [GRASS-dev] [GRASS-user] Keeping GRASS/OTB/… algorithm in qgis processing
On Mon, Feb 5, 2018 at 2:49 PM, Moritz Lennert <mlennert@club.worldonline.be> wrote:
On 05/02/18 13:51, Helmut Kudrnovsky wrote:
Von: “Moritz Lennert”
I don’t know how difficult it would be to create such algorithm
descriptions automagically.
https://github.com/qgis/QGIS/search?p=1&q=processing+grass&type=&utf8=%E2%9C%93
AFAIU there are some general python scripts to provide it in processing:
e.g.
python/plugins/processing/algs/grass7/Grass7AlgorithmProvider.py
No, this is the provider itself, not a tool to create the descriptions.
and there are a lot of txt files providing the module interface
e.g
python/plugins/processing/algs/grass7/description/r.out.png.txt
r.out.png
Export a GRASS raster map as a non-georeferenced PNG image
Raster (r.*)
QgsProcessingParameterRasterLayer|input|Input raster|None|False
QgsProcessingParameterNumber|compression|Compression level of PNG file (0 = none, 1 = fastest, 9 = best)|QgsProcessingParameterNumber.Integer|6|True|0|9
and other files
My question was whether it would be possible to create these description files more or less automagically.
I think the python scripts for more complex operations have to be created manually.
IIUC (from rapid reading of the threads on the qgis-developer list), Rashad’s suggestion was to keep the *AlgorithmProvider code in the QGIS code base, but to possibly move the creation of the description and script files to a plugin managed outside QGIS core, possibly by the respective external software teams.
Yes. your are right on track! the idea is external tools (processing providers) manage descriptor files in a format requested by qgis processing.
I see already name-of-grass-module --interface-descriptor which gives an xml for GRASS gui.
what qgis want is a csv in a specific format. The contents of qgis descriptor seems much less compared to --interface-descriptor.
Correct me if I am wrong, --interface-descriptor is available in all grass modules. So maybe a --qgis can do the work.
This has some advantages.
-
GRASS developers are free to fix parameter name, parameter description, list of modules(add and remove) changes without affecting qgis.
-
grass 7.5 has 10 modules and grass 7.9 can have 15 and same QGIS will work.
-
QGIS will not need to maintain these files and keep updating/adding new modules with their release process. whatever is generated from a grass build/install have better integration with qgis
-
Finally this descriptors for qgis are generated with a makefile target that allows users and packagers to include it.
-
QGIS can use multiple version of grass by changing install prefix because descriptors are always found in a directory relative to install prefix.
QGIS provider will be like:
I picked a descriptor file
parse and make the ui,
take input and execute whatever program the descriptor is to run.
QGIS already manages parsing of parameters and running them. So for providers who wish to be integrated in qgis will deal with a descriptor file and things go fine.
Having a single interface to launch all or most of toolboxes (willing to contribute descriptor with installation) can have same way of execution. At that point, QGIS should consider
adding some generic code in provider and avoid plugins for such toolboxes.
Moritz
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev
–
Regards,
Rashad
grass-user mailing list
grass-user@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-user