[GRASS-dev] Adding an expert mode to the parser

GRASS GRASS developer
1

Hi,

in order to not hinder the addition of important new flags/parameters
to modules while keeping things optionally easy it would be good to
implement some expert mode to the parser.

The flags and parameters for advanced users should be hidden by
default (maybe by using an extra definition in order to "tag" them in
the source code). Then e.g. by setting a variable they would become
visible in the help text and GUI.
Probably they should be always accepted when being invoked.

Any thoughts if such mechanism would be complicated to be implemented?

Markus

2

On Thu, Sep 22, 2016 at 6:00 PM, Markus Neteler <neteler@osgeo.org> wrote:

Hi,

in order to not hinder the addition of important new flags/parameters
to modules while keeping things optionally easy it would be good to
implement some expert mode to the parser.

The flags and parameters for advanced users should be hidden by
default (maybe by using an extra definition in order to "tag" them in
the source code). Then e.g. by setting a variable they would become
visible in the help text and GUI.
Probably they should be always accepted when being invoked.

Any thoughts if such mechanism would be complicated to be implemented?

Probably not so complicated, but I would be more concerned with the
decisions coming with this. For example, what should the online manual
show? How do we decide which options are advanced. How do we make sure
people understand the options were not removed, but are just hidden?

If the goal of this is to simplify life for new users, we can focus
the efforts on GUI. If we tag an option as advanced in parser, it
could be moved to a special tab 'Advanced', which some modules already
have. I am not sure hiding the options completely is a good idea. Also
in manual pages, perhaps these advanced options could somehow be
visibly marked as advanced (or possibly hidden with some javascript?).
Let's see what others think about this.

Anna

Markus
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

3

On 23/09/16 02:37, Anna Petrášová wrote:

On Thu, Sep 22, 2016 at 6:00 PM, Markus Neteler <neteler@osgeo.org> wrote:

Hi,

in order to not hinder the addition of important new flags/parameters
to modules while keeping things optionally easy it would be good to
implement some expert mode to the parser.

The flags and parameters for advanced users should be hidden by
default (maybe by using an extra definition in order to "tag" them in
the source code). Then e.g. by setting a variable they would become
visible in the help text and GUI.
Probably they should be always accepted when being invoked.

Any thoughts if such mechanism would be complicated to be implemented?

Probably not so complicated, but I would be more concerned with the
decisions coming with this. For example, what should the online manual
show? How do we decide which options are advanced. How do we make sure
people understand the options were not removed, but are just hidden?

If the goal of this is to simplify life for new users, we can focus
the efforts on GUI. If we tag an option as advanced in parser, it
could be moved to a special tab 'Advanced', which some modules already
have. I am not sure hiding the options completely is a good idea. Also
in manual pages, perhaps these advanced options could somehow be
visibly marked as advanced (or possibly hidden with some javascript?).
Let's see what others think about this.

If any, I would say +1 for Anna's suggestion. I don't really like the idea of "hiding" options behind an environment variable.

This said, I haven't really experienced that many problems with complexity of modules with newbies. Yes, some are a bit overwhelmed at first contact, but with a little guidance they quickly get the hang of things...

Moritz

4

On Fri, Sep 23, 2016 at 10:15 AM, Moritz Lennert <mlennert@club.worldonline.be> wrote:

On 23/09/16 02:37, Anna Petrášová wrote:
…

Probably not so complicated, but I would be more concerned with the
decisions coming with this. For example, what should the online manual
show?

Using the parser, the advanced options could automatically go into a subsection of the description part.
Just to distinguish them easily from the “standard” options.

How do we decide which options are advanced.

We simply keep 95% of all as-is.
But those we need for example for improved HPC processing (yet to be added) we may not want to clutter the standard interface with,

How do we make sure people understand the options were not
removed, but are just hidden?

Easy: since they show up in the manual, it is documented.
Having an “ advanced” button in the GUI, it is obvious.
For those being on command line, we could either show the advanced options, or, if not show instead a hint that this particular command also offers advanced options.

If the goal of this is to simplify life for new users, we can focus
the efforts on GUI.

Yes, that’s easy. But I am indeed thinking about the command line here.

If we tag an option as advanced in parser, it
could be moved to a special tab ‘Advanced’, which some modules already
have.

Yes.

I am not sure hiding the options completely is a good idea.

This is not what I intended (nor wrote I think).

Also in manual pages, perhaps these advanced options could somehow be
visibly marked as advanced (or possibly hidden with some javascript?).

Exactly.

If any, I would say +1 for Anna’s suggestion. I don’t really like the idea
of “hiding” options behind an environment variable.

It is not about hiding, as I just tried to explain maybe better now.
It is about not adding so many more flags and parameters to the standard interface, or, a way how to address this upcoming problem for the near future.

This said, I haven’t really experienced that many problems with complexity
of modules with newbies. Yes, some are a bit overwhelmed at first contact,
but with a little guidance they quickly get the hang of things…

But extrapolating from the '90th, we tend to add more and more…

Example: today d.vect comes with this mount of flags and parameters:
d.vect --interface-description | grep ‘flag name|parameter name’ | wc -l
39

In a single line (dirty hack):

for i in ls -1 bin/* scripts/* ; do echo -n “$i,” ; $i --interface-description | grep ‘flag name|parameter name’ | wc -l ; done | sort -n -t’,’ -k2
[…]
bin/r.watershed,27
bin/r.sim.water,28
bin/v.in.ogr,28
bin/r.gwflow,29
bin/v.vol.rst,29
bin/r.solute.transport,32
bin/v.surf.rst,32
bin/d.legend,33
bin/r.sun,34
bin/d.vect,39
bin/g.region,40

So, would adding 25% of new flags and parameters to many modules in the next years still be ok? I guess not. So we need to understand how to handle that.
Hence my proposal to structure them a bit better.

cheers,
Markus

5

On 23/09/16 13:11, Markus Neteler wrote:

On Fri, Sep 23, 2016 at 10:15 AM, Moritz Lennert
<mlennert@club.worldonline.be <mailto:mlennert@club.worldonline.be>> wrote:

On 23/09/16 02:37, Anna Petrášová wrote:

...

Probably not so complicated, but I would be more concerned with the
decisions coming with this. For example, what should the online manual
show?

Using the parser, the advanced options could automatically go into a
subsection of the description part.
Just to distinguish them easily from the "standard" options.

How do we decide which options are advanced.

We simply keep 95% of all as-is.
But those we need for example for improved HPC processing (yet to be
added) we may not want to clutter the standard interface with,

No possibility of embedding this into some configuration module (à la r.li.setup) which sets the relevant options and then just adding one flag to the other modules, e.g. '-h' for 'take into account the HPC settings ?

How do we make sure people understand the options were not
removed, but are just hidden?

Easy: since they show up in the manual, it is documented.
Having an " advanced" button in the GUI, it is obvious.
For those being on command line, we could either show the advanced
options, or, if not show instead a hint that this particular command
also offers advanced options.

I would plead for a separate, but visible 'advanced options' section.

Example: today d.vect comes with this mount of flags and parameters:
d.vect --interface-description | grep 'flag name\|parameter name' | wc -l
39

In a single line (dirty hack):

for i in `ls -1 bin/* scripts/*` ; do echo -n "$i," ; $i
--interface-description | grep 'flag name\|parameter name' | wc -l ;
done | sort -n -t',' -k2
[...]
bin/r.watershed,27
bin/r.sim.water,28
bin/v.in.ogr,28
bin/r.gwflow,29
bin/v.vol.rst,29
bin/r.solute.transport,32
bin/v.surf.rst,32
bin/d.legend,33
bin/r.sun,34
bin/d.vect,39
bin/g.region,40

So, would adding 25% of new flags and parameters to many modules in the
next years still be ok? I guess not. So we need to understand how to
handle that.
Hence my proposal to structure them a bit better.

Ok. We might also think about the option of subdividing those modules with too many flags into several separate modules sharing the same code base. Or go the QGIS way, i.e. create several pre-configured wrapper scripts for the most "complicated" modules with several options preset and thus not visible in the interface.

But, I see your point and it is definitely necessary to think about the options.

Moritz

6

On Fri, Sep 23, 2016 at 1:11 PM, Markus Neteler <neteler@osgeo.org> wrote:

On Fri, Sep 23, 2016 at 10:15 AM, Moritz Lennert
<mlennert@club.worldonline.be> wrote:

On 23/09/16 02:37, Anna Petrášová wrote:

...

How do we decide which options are advanced.

We simply keep 95% of all as-is.
But those we need for example for improved HPC processing (yet to be added)
we may not want to clutter the standard interface with,

Your motivation is to provide a specialized CLI interface for HPC
processing? We used GRASS with HPC processing for years and the
problems we faced were causes by the HPC hardware and software
infrastructure, not by GRASS. What exactly is the problem with using
GRASS and HPC processing? You could ask on the GRASS user ML, or a
GRASS developer who has experience with HPC processing.

How do we make sure people understand the options were not
removed, but are just hidden?

For those being on command line, we could either show the advanced options,
or, if not show instead a hint that this particular command also offers
advanced options.

I assume that most GRASS users regard the command line as advanced
usage, therefore the CLI should always show all flags and options.

If the goal of this is to simplify life for new users, we can focus
the efforts on GUI.

Yes, that's easy. But I am indeed thinking about the command line here.

See above. Why exactly do you want to change the command line
interface? What is the problem?

If we tag an option as advanced in parser, it
could be moved to a special tab 'Advanced', which some modules already
have.

Yes.

I am not sure hiding the options completely is a good idea.

This is not what I intended (nor wrote I think).

You wrote "The flags and parameters for advanced users should be
hidden by default", and Anna referred to such advanced options.

It is not about hiding, as I just tried to explain maybe better now.
It is about not adding so many more flags and parameters to the standard
interface, or, a way how to address this upcoming problem for the near
future.

As you said, your problem is related to HPC processing. Why do you
think more flags and options are a problem for HPC processing?
Particularly for HPC processing, I would assume that the more
information and the more control you have, the better.

Markus M

7

On Fri, Sep 23, 2016 at 11:05 PM, Markus Metz
<markus.metz.giswork@gmail.com> wrote:

On Fri, Sep 23, 2016 at 1:11 PM, Markus Neteler <neteler@osgeo.org> wrote:

...

Your motivation is to provide a specialized CLI interface for HPC
processing?

No, not the case.

We used GRASS with HPC processing for years and the
problems we faced were causes by the HPC hardware and software
infrastructure, not by GRASS. What exactly is the problem with using
GRASS and HPC processing?

There is no problem. There is just the issue that with an increasing
amount of additions (e.g. maybe the need to provide region/resolution
to individual modules for independent parallel processing without the
overhead of always opening a new mapset) to the user interface it
becomes harder to understand for most users.

You could ask on the GRASS user ML, or a
GRASS developer who has experience with HPC processing.

I have 10+ years of experience in GRASS HPC :slight_smile: So that's fine.

How do we make sure people understand the options were not
removed, but are just hidden?

For those being on command line, we could either show the advanced options,
or, if not show instead a hint that this particular command also offers
advanced options.

I assume that most GRASS users regard the command line as advanced
usage, therefore the CLI should always show all flags and options.

I don't think that all CLI users want to see always 40+ flags and
options per command if it could also be customizable.
Of course there may be different opinions.

...

You wrote "The flags and parameters for advanced users should be
hidden by default", and Anna referred to such advanced options.

Simple example, not HPC related:

* r.sun: "npartitions" - not really relevant for day2day "normal" usage.
* various commands: "Use the low-memory version" - also more advanced
etc.

It is not about hiding, as I just tried to explain maybe better now.
It is about not adding so many more flags and parameters to the standard
interface, or, a way how to address this upcoming problem for the near
future.

As you said, your problem is related to HPC processing. Why do you
think more flags and options are a problem for HPC processing?

No, not a problem for using it.
But maybe not interesting for all users to be always seen when using
GRASS in "shallow mode".

Particularly for HPC processing, I would assume that the more
information and the more control you have, the better.

Exactly. Just not all (new) users want to see a huge interface of
which they only use a fraction.
That's why it could be structured better in order to distinguish
between "easy" and "advanced".
Maybe not hide then but at least group the parameters and flags rather
than alpha-order or random mix.

markusN

8

On Fri, Sep 23, 2016 at 11:22 PM, Markus Neteler <neteler@osgeo.org> wrote:

On Fri, Sep 23, 2016 at 11:05 PM, Markus Metz
<markus.metz.giswork@gmail.com> wrote:

On Fri, Sep 23, 2016 at 1:11 PM, Markus Neteler <neteler@osgeo.org> wrote:

...

Your motivation is to provide a specialized CLI interface for HPC
processing?

No, not the case.

We used GRASS with HPC processing for years and the
problems we faced were causes by the HPC hardware and software
infrastructure, not by GRASS. What exactly is the problem with using
GRASS and HPC processing?

There is no problem. There is just the issue that with an increasing
amount of additions (e.g. maybe the need to provide region/resolution
to individual modules for independent parallel processing without the
overhead of always opening a new mapset)

Getting closer it seems. Can you quantify "the overhead of always
opening a new mapset"?

Markus M

9

On Fri, Sep 23, 2016 at 11:30 PM, Markus Metz
<markus.metz.giswork@gmail.com> wrote:

On Fri, Sep 23, 2016 at 11:22 PM, Markus Neteler <neteler@osgeo.org> wrote:

On Fri, Sep 23, 2016 at 11:05 PM, Markus Metz
<markus.metz.giswork@gmail.com> wrote:

On Fri, Sep 23, 2016 at 1:11 PM, Markus Neteler <neteler@osgeo.org> wrote:

...

Your motivation is to provide a specialized CLI interface for HPC
processing?

No, not the case.

We used GRASS with HPC processing for years and the
problems we faced were causes by the HPC hardware and software
infrastructure, not by GRASS. What exactly is the problem with using
GRASS and HPC processing?

There is no problem. There is just the issue that with an increasing
amount of additions (e.g. maybe the need to provide region/resolution
to individual modules for independent parallel processing without the
overhead of always opening a new mapset)

Getting closer it seems. Can you quantify "the overhead of always
opening a new mapset"?

As an example, when aiming at processing all Sentinel-2 tiles
globally, we speak about currently 73000 scenes * up-to-16 tiles along
with global data, analysis on top of other global data is more complex
when doing each job in its own mapset and reintegrate it in a single
target mapset as if able to process then in parallel in one mapset by
simply specifying the respective region to the command of interest.
Yes, different from the current paradigm and not for G7.

But my original comment was targeted at the increasing number of
module parameters and how to handle that (with some new HPC related
idea as an example).

I'm fine to archive this question for now, it will likely come up
again in the future.

markusN

10

Hi,
Here an example of hidden region settings for modules utilized in the temporal framework:

Imagine you have a Landsat location that contains many thousand scenes (time series) from all over the world including all bands. Each band of the time series is managed in a specific space-time dataset. You may want to use the temporal raster algebra to compute the NDVI in parallel on all scenes at once. This is not possible at the moment, because of the current region settings. Except you can set the current region to cover the whole world, which is not preferable.

So the idea is to have options that will set the computational region for each process independently, when the process is invoked. Hence the raster computational window is set in the parser, so that the usual region get methods in the modules will work untouched. This avoids the current approach that is covering a single module call (like r.mapclalc, r.series, r.neighbors, … ) in several g.region and g.remove calls in a dedicated subprocess.

This option is only useful for parallel processing of raster layers with different spatial extents. A special feature for the temporal framework, that should not be exposed to all the users, only to those who implement massive parallel working temporal modules. Hence using the same approach as for WPS, GUI and HTML output:

r.mapclac --raster-region= --north= --south= --west= --east= --res= --ewres= --nsres= --vect-region --raster-align= …

Just my 2 cent
Sören

···

2016-09-25 20:49 GMT+01:00 Markus Neteler <neteler@osgeo.org>:

On Fri, Sep 23, 2016 at 11:30 PM, Markus Metz
<markus.metz.giswork@gmail.com> wrote:

On Fri, Sep 23, 2016 at 11:22 PM, Markus Neteler <neteler@osgeo.org> wrote:

On Fri, Sep 23, 2016 at 11:05 PM, Markus Metz
<markus.metz.giswork@gmail.com> wrote:

On Fri, Sep 23, 2016 at 1:11 PM, Markus Neteler <neteler@osgeo.org> wrote:
…
Your motivation is to provide a specialized CLI interface for HPC
processing?

No, not the case.

We used GRASS with HPC processing for years and the
problems we faced were causes by the HPC hardware and software
infrastructure, not by GRASS. What exactly is the problem with using
GRASS and HPC processing?

There is no problem. There is just the issue that with an increasing
amount of additions (e.g. maybe the need to provide region/resolution
to individual modules for independent parallel processing without the
overhead of always opening a new mapset)

Getting closer it seems. Can you quantify “the overhead of always
opening a new mapset”?

As an example, when aiming at processing all Sentinel-2 tiles
globally, we speak about currently 73000 scenes * up-to-16 tiles along
with global data, analysis on top of other global data is more complex
when doing each job in its own mapset and reintegrate it in a single
target mapset as if able to process then in parallel in one mapset by
simply specifying the respective region to the command of interest.
Yes, different from the current paradigm and not for G7.

But my original comment was targeted at the increasing number of
module parameters and how to handle that (with some new HPC related
idea as an example).

I’m fine to archive this question for now, it will likely come up
again in the future.

markusN

grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

11

[snip]

>r.mapclac --raster-region= --north= --south= --west= --east= --res=
>--ewres= --nsres= --vect-region --raster-align= ...

Can't you do this at a higher level, i.e using the GRASS Python API's
use_temp_region ?

This is exactly what i'm doing right now[1,2], covering a single module
call in several g.region and g.remove calls in a dedicated subprocess.
Huge, massive unnecessary overhead.

[1]
https://trac.osgeo.org/grass/browser/grass/trunk/lib/python/pygrass/modules/interface/module.py#L791
[2]
https://trac.osgeo.org/grass/browser/grass/trunk/lib/python/pygrass/modules/interface/module.py#L951

Moritz

>
>Just my 2 cent
>Sören
>
>2016-09-25 20:49 GMT+01:00 Markus Neteler <neteler@osgeo.org>:
>
>> On Fri, Sep 23, 2016 at 11:30 PM, Markus Metz
>> <markus.metz.giswork@gmail.com> wrote:
>> > On Fri, Sep 23, 2016 at 11:22 PM, Markus Neteler
><neteler@osgeo.org>
>> wrote:
>> >> On Fri, Sep 23, 2016 at 11:05 PM, Markus Metz
>> >> <markus.metz.giswork@gmail.com> wrote:
>> >>> On Fri, Sep 23, 2016 at 1:11 PM, Markus Neteler
><neteler@osgeo.org>
>> wrote:
>> >> ...
>> >>> Your motivation is to provide a specialized CLI interface for HPC
>> >>> processing?
>> >>
>> >> No, not the case.
>> >>
>> >>> We used GRASS with HPC processing for years and the
>> >>> problems we faced were causes by the HPC hardware and software
>> >>> infrastructure, not by GRASS. What exactly is the problem with
>using
>> >>> GRASS and HPC processing?
>> >>
>> >> There is no problem. There is just the issue that with an
>increasing
>> >> amount of additions (e.g. maybe the need to provide
>region/resolution
>> >> to individual modules for independent parallel processing without
>the
>> >> overhead of always opening a new mapset)
>> >
>> > Getting closer it seems. Can you quantify "the overhead of always
>> > opening a new mapset"?
>>
>> As an example, when aiming at processing all Sentinel-2 tiles
>> globally, we speak about currently 73000 scenes * up-to-16 tiles
>along
>> with global data, analysis on top of other global data is more
>complex
>> when doing each job in its own mapset and reintegrate it in a single
>> target mapset as if able to process then in parallel in one mapset by
>> simply specifying the respective region to the command of interest.
>> Yes, different from the current paradigm and not for G7.
>>
>> But my original comment was targeted at the increasing number of
>> module parameters and how to handle that (with some new HPC related
>> idea as an example).
>>
>> I'm fine to archive this question for now, it will likely come up
>> again in the future.
>>
>> markusN
>> _______________________________________________
>> grass-dev mailing list
>> grass-dev@lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/grass-dev
>>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>grass-dev mailing list
>grass-dev@lists.osgeo.org
>http://lists.osgeo.org/mailman/listinfo/grass-dev

12

On Sun, Sep 25, 2016 at 5:40 PM, Sören Gebbert <soerengebbert@googlemail.com

wrote:

>r.mapclac --raster-region= --north= --south= --west= --east= --res=

>--ewres= --nsres= --vect-region --raster-align= ...

I like this in the sense that if the region setting for the module is
something which should be managed by these extra options, then they should
be managed in parser, rather than introduced one by one for individual
modules.

But how is this different from using GRASS_REGION? Convenience of
--raster-region=?
Better syntax than environmental variable?

13

On Sun, Sep 25, 2016 at 3:49 PM, Markus Neteler <neteler@osgeo.org> wrote:

when doing each job in its own mapset and reintegrate it in a single
target mapset as if able to process then in parallel in one mapset by
simply specifying the respective region to the command of interest.
Yes, different from the current paradigm and not for G7.

I think, we can accommodate this behavior in G7. In fact, each command can
run with a separate region even now. It can set it on its own, take it from
GRASS_REGION or WIND_OVERRIDE. But if you are talking about automatic
patching, that's of course different.

But my original comment was targeted at the increasing number of
module parameters and how to handle that (with some new HPC related
idea as an example).

I think we need to talk about specific use cases and decide based on that.
For some, Advanced or HPC tab might be enough. Some others might be better
addressed by a specific global options like what Soeren just suggested (in
the style of --overwrite).

14

Hi,

I have been struggling with similar issues and solved them by writing a GRASS_BATCH_JOB script that I execute in parallel in different mapsets.

Thus, the approach Markus and Sören sketch looks very useful to me (even for people who are not on HPC).

Do you have more options / flags in mind than the region settings? Just asking to get a better idea about the amount of how module complexity may increase with the changes you plan?

If it is only region settings per module-call that should not be overwhelming for users. E.g., the QGIS-GRASS plugin (and I guess Processing too) has a button for «use region of this map» for every command with raster input, which is pointing in this direction (though not as sophisticated as your solution). And QGIS aimed at simplifying the interaction with GRASS analysis ;-).

In order to not blowing up the manual too much, one might write a compact standard description that points to g.region or a central, more detailed description of the new options, which will be the same for every module that uses them, would not they?

After you explained a bit more what you are after I do not really see a problem with this. In fact I am very positive!

But I have to admit that I did not understand the difference between --raster-region and --vector-region. Are you planning to implement some vector-tiles solution?

Cheers

Stefan

···

On Sun, Sep 25, 2016 at 3:49 PM, Markus Neteler <neteler@osgeo.org> wrote:

when doing each job in its own mapset and reintegrate it in a single
target mapset as if able to process then in parallel in one mapset by
simply specifying the respective region to the command of interest.
Yes, different from the current paradigm and not for G7.

I think, we can accommodate this behavior in G7. In fact, each command can run with a separate region even now. It can set it on its own, take it from GRASS_REGION or WIND_OVERRIDE. But if you are talking about automatic patching, that’s of course different.

But my original comment was targeted at the increasing number of
module parameters and how to handle that (with some new HPC related
idea as an example).

I think we need to talk about specific use cases and decide based on that. For some, Advanced or HPC tab might be enough. Some others might be better addressed by a specific global options like what Soeren just suggested (in the style of --overwrite).

15

Hi,

···

On Fri, Sep 23, 2016 at 12:00 AM, Markus Neteler <neteler@osgeo.org> wrote:

Hi,

in order to not hinder the addition of important new flags/parameters
to modules while keeping things optionally easy it would be good to
implement some expert mode to the parser.

The flags and parameters for advanced users should be hidden by
default (maybe by using an extra definition in order to “tag” them in
the source code). Then e.g. by setting a variable they would become
visible in the help text and GUI.
Probably they should be always accepted when being invoked.

This makes sense to me. If I type name.module --help and the command line yield too long a list of parameters, I normally prefer reading it on the manual page, where it is more clear, and I find extremely useful the usage examples of the flags. I don’t find it very informative to list all the flags in command line, even the less used. There could be for example a --ext-help for extended help display. So for example, typing only --help would list only most used options and would end with “type --ext-help for extended help” or something like that.

My 2 c

–

Margherita Di Leo

16

Hello

I’m with MaDi in that if I see a very long list of flags and parameters in the terminal when using --h, i just go to the manual… I just use --h in CLI for a quick recalling of flags/options… A reduced list of most commonly used flags would be nice, but still keep the possibility to get the advanced flags/parameters as well, if the user needs more info… and the same in the GUI and manual pages then (a tab or section for advanced flags/parameters)…

IMHO, a major issue however is the lack of examples for the usage of more advanced flags or options (or even the required and more common ones). Take the case of several i.* modules, for example… but maybe this should go in a separate thread :slight_smile:

Cheers,
Vero

···

2016-09-26 6:01 GMT-03:00 Margherita Di Leo <diregola@gmail.com>:

Hi,

grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

On Fri, Sep 23, 2016 at 12:00 AM, Markus Neteler <neteler@osgeo.org> wrote:

Hi,

in order to not hinder the addition of important new flags/parameters
to modules while keeping things optionally easy it would be good to
implement some expert mode to the parser.

The flags and parameters for advanced users should be hidden by
default (maybe by using an extra definition in order to “tag” them in
the source code). Then e.g. by setting a variable they would become
visible in the help text and GUI.
Probably they should be always accepted when being invoked.

This makes sense to me. If I type name.module --help and the command line yield too long a list of parameters, I normally prefer reading it on the manual page, where it is more clear, and I find extremely useful the usage examples of the flags. I don’t find it very informative to list all the flags in command line, even the less used. There could be for example a --ext-help for extended help display. So for example, typing only --help would list only most used options and would end with “type --ext-help for extended help” or something like that.

My 2 c

–

Margherita Di Leo

17

On Mon, Sep 26, 2016 at 5:24 PM, Veronica Andreo <veroandreo@gmail.com>
wrote:

I'm with MaDi in that if I see a very long list of flags and parameters in
the terminal when using --h, i just go to the manual... I just use --h in
CLI for a quick recalling of flags/options... A reduced list of most
commonly used flags would be nice, but still keep the possibility to get
the advanced flags/parameters as well, if the user needs more info...

If the --help is just for scanning and the issue is that it simply too
long, hiding some parameters is not the only option we have. For many (and
more in the future hopefully) parameters we have (short) label and (longer)
description. --help prints both (if both are present, that's at least 2
lines per parameter). Additionally, if the option has predefined values
which have descriptions, there is one line for each of those. So, the
question is would it be helpful (at least as a first step) if --help would
print less information for each parameter but still provided all parameters?

and the same in the GUI

In the case of GUI, we should first answer the question why are the
sections/tabs not enough? Because they already should separate basic and
advanced while providing actually much finer and partially self-documenting
way (for example, you fill in Required tab, go also to Selection, but leave
aside Transformation and Advanced tabs).

and manual pages then (a tab or section for advanced flags/parameters)...

Considering that we have currently as system of (gui) sections which place
the options to individual tabs in GUI, isn't showing the different sections
in the manual the right thing to do? This would not only make the manual
more organized, but it would also link the manual more to the GUI.
Depending how good and smart are we with the styling and JavaScript we can
even hide some by default. But perhaps there is a reason why this is not
enough.

IMHO, a major issue however is the lack of examples for the usage of more
advanced flags or options (or even the required and more common ones). Take
the case of several i.* modules, for example... but maybe this should go in
a separate thread :slight_smile:

Good point. If you have an explanation and example for a flag, perhaps you
can understand it and it is not so advanced at the end.

18

On Fri, Sep 23, 2016 at 5:22 PM, Markus Neteler <neteler@osgeo.org> wrote:

Maybe not hide then but at least group the parameters and flags rather
than alpha-order or random mix.

This is something which can be done right now. The options and flags are
presented in the order as they are defined in the source code. These
additional rules currently apply in general: (one letter) flags are listed
before (key-value) options. Long (multi-letter standard) flags are listed
in between them. Additionally in GUI, before this sorting, the options and
flags are always sorted to the tabs according to the specified (or
automatically generated) (gui) sections. Typically options and flags
(separately or combined) with (gui) section are specified together in the
source code, consequently they are grouped in --help and manual (but
without visible borders or any other indication).

Of course, this doesn't mean that the (gui) section cannot be contain, for
example 'basic', 'common' and 'advanced' subsections.

19

On Fri, Sep 23, 2016 at 5:22 PM, Markus Neteler <neteler@osgeo.org> wrote:

* various commands: "Use the low-memory version" - also more advanced

I guess this is a typical example where we need to be careful. I can
imagine a situation when a beginner has a lot of data and low-end hardware.
By hiding an parameter like this, we would make it even more advanced. Now
you not only need to understand that your hardware is not enough and that
the algorithm should somehow account for that. You also need to know that
there might be a hidden option for that and you need to know how to find
it. Having this in the manual or even a Memory or Speed tab in the GUI,
gives some hope that even a beginner will be able to use it.

20

On 26/09/16 23:50, Vaclav Petras wrote:

On Mon, Sep 26, 2016 at 5:24 PM, Veronica Andreo <veroandreo@gmail.com
<mailto:veroandreo@gmail.com>> wrote:

    I'm with MaDi in that if I see a very long list of flags and
    parameters in the terminal when using --h, i just go to the
    manual... I just use --h in CLI for a quick recalling of
    flags/options... A reduced list of most commonly used flags would be
    nice, but still keep the possibility to get the advanced
    flags/parameters as well, if the user needs more info...

If the --help is just for scanning and the issue is that it simply too
long, hiding some parameters is not the only option we have. For many
(and more in the future hopefully) parameters we have (short) label and
(longer) description. --help prints both (if both are present, that's at
least 2 lines per parameter). Additionally, if the option has predefined
values which have descriptions, there is one line for each of those. So,
the question is would it be helpful (at least as a first step) if --help
would print less information for each parameter but still provided all
parameters?

In line with your other mail where you caution about "hiding" useful information, how about not changing the --help output, but rather adding a "--simple-help" or somthing like this which would output a simplified help. Although:

    and manual pages then (a tab or section for advanced
    flags/parameters)...

Considering that we have currently as system of (gui) sections which
place the options to individual tabs in GUI, isn't showing the different
sections in the manual the right thing to do?

I would prefer this: show everything, but structure it differently, i.e. possibly introduce a new parser keyword (advanced: yes) which would put the option into a specific section of the manual. IMHO, this should be independent of the GUI sections logic as one might to group less advanced and advanced options in the same thematic tab...

  IMHO, a major issue however is the lack of examples for the usage of
  more advanced flags or options (or even the required and more common
  ones). Take the case of several i.* modules, for example... but maybe
  this should go in a separate thread :slight_smile:

>
> Good point. If you have an explanation and example for a flag, perhaps
> you can understand it and it is not so advanced at the end.

I think this is actually a major issue: man pages without description, notes and example sections are almost useless IMHO. At the foss4g.be last week someone presented a simple use of GRASS GIS (to create this map [1] for television) and explained how he actually found the GRASS GIS manual system extremely well done and useful, because of the explanations and examples...

Moritz

[1] http://www.rtbf.be/services/meteo/apere