[GRASS-dev] [GRASS GIS] #2895: Define dependencies for GRASS addons

GDAL · February 5, 2016, 8:55am

#2895: Define dependencies for GRASS addons
-------------------------+-------------------------
Reporter: pmav99 | Owner: grass-dev@…
Type: enhancement | Status: new
Priority: normal | Milestone:
Component: Addons | Version: unspecified
Keywords: g.extension | CPU: Unspecified
Platform: Unspecified |
-------------------------+-------------------------
Some addons depend on other addons. E.g. `r.lfp` depends on
`r.stream.distance`.

If you install `r.lfp` and you try to use it while `r.stream.distance` is
not installed, you get the following traceback which is not particularly
helpful; especially so if you are not familiar with Python.
{{{
Traceback (most recent call last):
   File "/home/grassuser/.grass7/addons/scripts/r.lfp", line 111, in
<module>
     sys.exit(main())
   File "/home/grassuser/.grass7/addons/scripts/r.lfp", line 43, in main
     calculate_lfp(input, output, coords)
   File "/home/grassuser/.grass7/addons/scripts/r.lfp", line 67, in
calculate_lfp
     distance=flds)
   File "/usr/lib/grass70/etc/python/grass/script/core.py", line 392, in
run_command
     ps = start_command(*args, **kwargs)
   File "/usr/lib/grass70/etc/python/grass/script/core.py", line 361, in
start_command
     return Popen(args, **popts)
   File "/usr/lib/grass70/etc/python/grass/script/core.py", line 64, in
__init__
     subprocess.Popen.__init__(self, args, **kwargs)
   File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
     errread, errwrite)
   File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
     raise child_exception
OSError: [Errno 2] No such file or directory
}}}
There should be a way to define GRASS dependencies. E.g. something like
`requirements.txt` that is being used by Python packages.

Optimally `g.extension` should read the list of depedencies and install
missing ones automatically. If `g.extension` does go down this road, then
an additional flag for skipping dependency checks might also be a good
idea.

Perhaps this could/should also be expanded to additional non-GRASS related
dependencies. E.g. `r.denoise` requires the existence of the `mdenoise`
binary. In this case IMHO a graceful failure message would be preferable
to a successful installation of a non-working addon.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2895>
GRASS GIS <https://grass.osgeo.org>

GDAL · February 5, 2016, 9:16am

#2895: Define dependencies for GRASS addons
--------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone:
Component: Addons | Version: unspecified
Resolution: | Keywords: g.extension
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------

Comment (by pmav99):

Transferring Helmut Kudrnovsky's answer here:

> have a look here:
> https://trac.osgeo.org/grass/browser/grass-
addons/grass7/raster/r.basin/r.basin.py#L82
> {{{#!python
> # check requirements
> def check_progs():
> found_missing = False
> for prog in ('r.hypso', 'r.stream.basins', 'r.stream.distance',
'r.stream.extract',
> 'r.stream.order','r.stream.snap','r.stream.stats', 'r.width.funct'):
> if not grass.find_program(prog, '--help'):
> found_missing = True
> grass.warning(_("'%s' required. Please install '%s' first
using 'g.extension %s'") % (prog, prog, prog))
> if found_missing:
> grass.fatal(_("An ERROR occurred running r.basin"))
> }}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2895#comment:1>
GRASS GIS <https://grass.osgeo.org>

GDAL · February 5, 2016, 10:32am

#2895: Define dependencies for GRASS addons
--------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.0.4
Component: Addons | Version: unspecified
Resolution: | Keywords: g.extension
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------
Changes (by martinl):

* milestone: => 7.0.4

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2895#comment:2>
GRASS GIS <https://grass.osgeo.org>

GDAL · May 5, 2019, 7:27pm

#2895: Define dependencies for GRASS addons
--------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone:
Component: Addons | Version: unspecified
Resolution: | Keywords: g.extension
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------

Comment (by neteler):

It might be a good idea to implement some function(s) for dependency
lookup from a module-specific requirements.txt - I see some (randomly
seeked) code snippet here:

https://stackoverflow.com/a/16624700/452464

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2895#comment:12>
GRASS GIS <https://grass.osgeo.org>

GDAL · May 5, 2019, 7:33pm

#2895: Define dependencies for GRASS addons
--------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Addons | Version: unspecified
Resolution: | Keywords: g.extension
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------
Changes (by martinl):

* milestone: => 7.8.0

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2895#comment:13>
GRASS GIS <https://grass.osgeo.org>

GDAL · May 5, 2019, 7:34pm

#2895: Define dependencies for GRASS addons
--------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Default | Version: unspecified
Resolution: | Keywords: g.extension
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------
Changes (by martinl):

* component: Addons => Default

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2895#comment:14>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 17, 2020, 8:30am

#2895: Define dependencies for GRASS addons
--------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.8.3
Component: Default | Version: unspecified
Resolution: | Keywords: g.extension
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------

Comment (by sbl):

Pyhon modules are supposed to use lazy imports and should catch if a
required python library is available at runtime. The same should be true
for dependency on other addons (should probably added to the SUBMITTING
guidelines).

For C modules, installation of libraries through g.extension is probably
not very straight forward.

So the question is a bit, where this is supposed to be handeled.
Personally, I would probably prefer if the addon itself catches evtl.
missing dependencies and gives a clear error message. Silently installing
stuff can be a bit scary...

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2895#comment:19>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 18, 2020, 2:14am

#2895: Define dependencies for GRASS addons
--------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.8.3
Component: Default | Version: unspecified
Resolution: | Keywords: g.extension
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------

Comment (by wenzeslaus):

Replying to [comment:19 sbl]:
> Pyhon modules are supposed to use lazy imports and should catch if a
required python library is available at runtime.

+1

> For C modules, installation of libraries through g.extension is probably
not very straight forward.

Right. For some addons, we actually have that as an optional dependency in
the core. However, pip, R, and conda can actually do that, so perhaps if
we use them in background, they could take care of that.

There have been suggestions like this in the past, but a lot of changed in
past years. pip is now basically part of Python. conda is quite common.
There is a lot of R packages using C++. For example //itzi// is using pip
rather than g.extension.

> Silently installing stuff can be a bit scary...

Why do you think it is scary for GRASS GIS? pip, R, conda, apt, yum, ...
all install dependencies of a package you asked for. Why this should not
happen for GRASS GIS?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2895#comment:20>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 18, 2020, 8:57am

#2895: Define dependencies for GRASS addons
--------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.8.3
Component: Default | Version: unspecified
Resolution: | Keywords: g.extension
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------

Comment (by sbl):

Replying to [comment:20 wenzeslaus]:
> > Silently installing stuff can be a bit scary...
>
> Why do you think it is scary for GRASS GIS? pip, R, conda, apt, yum, ...
all install dependencies of a package you asked for. Why this should not
happen for GRASS GIS?

Yes, you are right.

So, how would we setup a requirements.txt?

Should it contain different sections, like e.g.
{{{
GRASS_addons
r.area

R_packages
ggplot2

Python_libraries
rpy2>=1.1

cmd_tools
iconv
cs2cs

libraries
libgdal-grass
}}}

Cause the way these dependencies are installed varies a bit, with a
dpendency_check function for each of them...

Maybe better with a dependency json:

{{{
{
"GRASS_addons": [
     {
       "name": "r.area"
     }],
"Python_libraries": [
     {
       "name": "rpy2",
       "version": "1.1"
       "version_check": ">="
     }], ...
}
}}}

In the functions we might have to think about OS specific aspects (package
manager (incl. conda vs. pip), library names) as well as maybe versions
(e.g. numpy>=1.17).

Also, should a failed dependency check block the installation? Because
that it is not a trivial task I would opt for a warning rather than an
error message in case of missing or unresolved dependencies...

Lastly, should a dependency check function go into ''g.extension'' or into
the python library (script or pygrass). In the python lib it could be used
by AddOn devs at runtime (e.g. if it takes a json formated string or
dict)...

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2895#comment:21>
GRASS GIS <https://grass.osgeo.org>

mlennert · December 18, 2020, 11:18am

Am 18. Dezember 2020 09:57:53 MEZ schrieb GRASS GIS <trac@osgeo.org>:

#2895: Define dependencies for GRASS addons
--------------------------+-------------------------
Reporter: pmav99 | Owner: grass-dev@…
Type: enhancement | Status: new
Priority: normal | Milestone: 7.8.3
Component: Default | Version: unspecified
Resolution: | Keywords: g.extension
CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------

Comment (by sbl):

Replying to [comment:20 wenzeslaus]:
> > Silently installing stuff can be a bit scary...
>
> Why do you think it is scary for GRASS GIS? pip, R, conda, apt, yum, ...
all install dependencies of a package you asked for. Why this should not
happen for GRASS GIS?

I would personally plead for the automatic installation of dependencies only as optional. I always prefer to install dependencies via my distribution's package manager instead of automatic installation via tools such as conda, pip or others. And I don't think it realistic to take into account all different package management systems within GRASS GIS.

So, +1 to a dependency check accompanied by the option to install the missing dependencies, but not automatic installation per se.

Yes, you are right.

So, how would we setup a requirements.txt?

Should it contain different sections, like e.g.
{{{
GRASS_addons
    r.area

R_packages
    ggplot2

Python_libraries
    rpy2>=1.1

cmd_tools
    iconv
    cs2cs

libraries
    libgdal-grass
}}}

Cause the way these dependencies are installed varies a bit, with a
dpendency_check function for each of them...

Maybe better with a dependency json:

{{{
{
"GRASS_addons": [
    {
      "name": "r.area"
    }],
"Python_libraries": [
    {
      "name": "rpy2",
      "version": "1.1"
      "version_check": ">="
    }], ...
}
}}}

In the functions we might have to think about OS specific aspects (package
manager (incl. conda vs. pip), library names) as well as maybe versions
(e.g. numpy>=1.17).

GDAL · December 19, 2020, 11:11am

#2895: Define dependencies for GRASS addons
--------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.8.3
Component: Default | Version: unspecified
Resolution: | Keywords: g.extension
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------

Comment (by neteler):

In a simple way, I have defined the needed requirements of `t.rast.mosaic`
addon as a small shell script:

https://github.com/mundialis/t.rast.mosaic/blob/main/requirements.sh

Of course it would be better if `g.extension` could take care...

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2895#comment:22>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 29, 2020, 9:49am

#2895: Define dependencies for GRASS addons
--------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.8.3
Component: Default | Version: unspecified
Resolution: | Keywords: g.extension
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------
Changes (by sbl):

* Attachment "handle_dependencies.py" added.

Draft for a Python function to handle dependencies in g.extension

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2895>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 29, 2020, 10:02am

#2895: Define dependencies for GRASS addons
--------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.8.3
Component: Default | Version: unspecified
Resolution: | Keywords: g.extension
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------

Comment (by sbl):

I just added a draft for a python script that could take care of
dependencies in Python (conda not tested) and R packages. Other
dependencies (e.g. C-libraries, commandline tools) are just checked,
neither loading of the libraries not installation is supported at the
moment (not sure if the latter is realistic).
The function has three modes: check (warns of missing dependencies),
install (installs missing dependencies (if possible), abort (stops if
dependencies are missing).
It takes the following arguments as input:

  - dependency_type
  - dependency
  - version=None
  - version_comparison=None
  - repository=None
  - optional=False

and could be fed e.g. from a table describing dependencies. dependency
versions, and specific repositories are not supported yet. Let me know
what you think, and i see how it could be integrated in g.extension.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2895#comment:23>
GRASS GIS <https://grass.osgeo.org>

GDAL · December 31, 2020, 4:11pm

#2895: Define dependencies for GRASS addons
--------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.8.3
Component: Default | Version: unspecified
Resolution: | Keywords: g.extension
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------

Comment (by hcho):

Replying to [comment:23 sbl]:
> I just added a draft for a python script that could take care of
dependencies in Python (conda not tested) and R packages. Other
dependencies (e.g. C-libraries, commandline tools) are just checked,
neither loading of the libraries not installation is supported at the
moment (not sure if the latter is realistic).
> The function has three modes: check (warns of missing dependencies),
install (installs missing dependencies (if possible), abort (stops if
dependencies are missing).
> It takes the following arguments as input:
>
> - dependency_type
> - dependency
> - version=None
> - version_comparison=None
> - repository=None
> - optional=False
>
> and could be fed e.g. from a table describing dependencies. dependency
versions, and specific repositories are not supported yet. Let me know
what you think, and i see how it could be integrated in g.extension.

handle_dependencies.py looks good. Just have a few comments.

{{{
cmd R 3.4 >=
cmd cmdfail 3.4 >=
R_package igraph 0.7.1 >=
R_package R_fail_test 0.7.1 >=
}}}
looks unnatural and error-prone. Would it be possible to change this
format to
{{{
cmd R >= 3.4
cmd cmdfail >= 3.4
R_package igraph >= 0.7.1
R_package R_fail_test >= 0.7.1
}}}

Also, how about defining dependency information inside modules themselves
instead of using an external file? We already have `G_option_*()`
functions to handle option dependency. Maybe, `G_module_requires(void
*first, ...), G_module_requires_python(void *first, ...),
G_module_requires_r(void *first, ...)` and
{{{
G_module_requires("r.stream.distance", NULL);
G_module_requires_python("numpy", "gdal", NULL);
}}}
Then, add a new global flag `--dependencies` to spit out dependency
information?

Just my 2 cents.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2895#comment:24>
GRASS GIS <https://grass.osgeo.org>

GDAL · March 13, 2021, 4:06am

#2895: Define dependencies for GRASS addons
--------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.8.3
Component: Default | Version: unspecified
Resolution: | Keywords: g.extension
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------

Comment (by wenzeslaus):

Replying to [comment:23 sbl]:
> I just added a draft for a python script that could take care of
dependencies in Python (conda not tested) and R packages. Other
dependencies (e.g. C-libraries, commandline tools) are just checked,
neither loading of the libraries not installation is supported at the
moment (not sure if the latter is realistic).

The current code looks straightforward and perhaps worth trying it in
action (you can make an addon module which installs dependencies and then
runs g.extension on the actual module).

However, I think this can also get really complex. What about turning this
the other way around and focusing supporting GRASS modules which are
Python packages installed with conda or pip like itzi? They would just use
the existing systems for dependencies. Still, g.extension could use
handle_dependencies.py logic, but it would use it to install the module
rather than the dependencies.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2895#comment:25>
GRASS GIS <https://grass.osgeo.org>

GDAL · March 13, 2021, 4:23am

#2895: Define dependencies for GRASS addons
--------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.8.3
Component: Default | Version: unspecified
Resolution: | Keywords: g.extension
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------

Comment (by wenzeslaus):

Replying to [comment:24 hcho]:
> Replying to [comment:23 sbl]:
> {{{
> cmd R 3.4 >=
> cmd cmdfail 3.4 >=
> R_package igraph 0.7.1 >=
> R_package R_fail_test 0.7.1 >=
> }}}
> looks unnatural and error-prone. Would it be possible to change this
format to
> {{{
> cmd R >= 3.4
> cmd cmdfail >= 3.4
> R_package igraph >= 0.7.1
> R_package R_fail_test >= 0.7.1
> }}}

Instead of adding another dependency format which is a custom format, I
would suggest at least using an existing general format, e.g., JSON.
However, going a step further might be even better, Conda has a somewhat
general dependency file format (environment.yml) or, alternatively, tools
like [https://mybinder.readthedocs.io/en/latest/using/config_files.html
Binder] take advantage of existing dependency formats, i.e., use
requirements.txt for Python, DESCRIPTION for R, environment.yml for Conda,
etc. When you consider setup.py for Python, this transitions nicely to my
suggestion about supporting modules which are packages.

> Also, how about defining dependency information inside modules
themselves instead of using an external file? We already have
`G_option_*()` functions to handle option dependency. Maybe,
`G_module_requires(void *first, ...), G_module_requires_python(void
*first, ...), G_module_requires_r(void *first, ...)` ... Then, add a new
global flag `--dependencies` to spit out dependency information?

In this case, you would have to compile and run the module before figuring
out the dependencies. This would not be possible for C/C++ and it still
insist on lazy imports. Even if we say these two issues are not bothering
us in the end given other issues, this would be extra confusing since it
is exactly the opposite of what any other dependency/packaging system is
doing.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2895#comment:26>
GRASS GIS <https://grass.osgeo.org>

GDAL · March 13, 2021, 4:28am

#2895: Define dependencies for GRASS addons
--------------------------+-------------------------
  Reporter: pmav99 | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.8.3
Component: Default | Version: unspecified
Resolution: | Keywords: g.extension
       CPU: Unspecified | Platform: Unspecified
--------------------------+-------------------------

Comment (by wenzeslaus):

Replying to [comment:21 sbl]:
> Lastly, should a dependency check function go into ''g.extension'' or
into the python library...

g.extension can call it, but it should be in the library - length of
g.extension.py being one reason.

> ...(script or pygrass)

This looks like a new subpackage of grass. Python and R use some names
already: setuptools, distutils, devtools, ...

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2895#comment:27>
GRASS GIS <https://grass.osgeo.org>