[GRASS-dev] Python scripts in GRASS GIS: conditional loading of external libraries?

GRASS GRASS developer
1

Hi,

we are currently writing a Python script which needs to import some
heavy external libraries.
AFAIK the script is read twice, once by the Python interpreter, then
when having reached the grass.parser() line again in order to re-read
properly the
#% options ...
etc. statements.

Now: could the import of the external library (it takes in the range
of > 2min) be conditionalized to only be read at the second "run"?

I was searching in the g.parser code but didn't really find anything useful.
An example would also be fine (addon?).

Or am I on the wrong track?

thanks
Markus

2

Hi,

pá 19. 10. 2018 v 16:45 odesílatel Markus Neteler <neteler@osgeo.org> napsal:

we are currently writing a Python script which needs to import some
heavy external libraries.

if I understand well, a lazy import could help

"""
def main:
    from mylib import xyz
    ...

   return 0

if __name__ == "__main__":
    opt, flg = grass.parser()
    sys.exit(main())
"""

Martin

--
Martin Landa
http://geo.fsv.cvut.cz/gwiki/Landa
http://gismentors.cz/mentors/landa

3

* Markus Neteler <neteler@osgeo.org> [2018-10-19 16:45:29 +0200]:

Hi,

we are currently writing a Python script which needs to import some
heavy external libraries.
AFAIK the script is read twice, once by the Python interpreter, then
when having reached the grass.parser() line again in order to re-read
properly the
#% options ...
etc. statements.

Now: could the import of the external library (it takes in the range
of > 2min) be conditionalized to only be read at the second "run"?

I was searching in the g.parser code but didn't really find anything useful.
An example would also be fine (addon?).

Or am I on the wrong track?

An idea is to try make use of `importlib.import_module()` along with
`sys`, like:

import sys
import importlib
if 'xyz' not in sys.modules:
    importlib.import_module('xyz')

or better (?):

if 'xyz' not in sys.modules and 'xyz' not in dir():
    importlib.import_module('xyz')

Although this is a valid test, according to some answers over at
StackExchange, I still miss something to make it work.

In ipython, the import works, but I am not able to access interactively
'xyz.' methods though.

Note to self: understand why.

Nikos

4

On Fri, Oct 19, 2018 at 12:23 PM Nikos Alexandris <nik@nikosalexandris.net> wrote:

Now: could the import of the external library (it takes in the range
of > 2min) be conditionalized to only be read at the second “run”?

I was searching in the g.parser code but didn’t really find anything useful.
An example would also be fine (addon?).

Or am I on the wrong track?

This is a Python thing, so g.parser doesn’t know anything about that. As Martin suggests, any import happening after the grass.script.parser() function call are good and lazy enough (i.e. after the

There is many examples of this already. Especially relevant examples are in addons where the point is exactly to avoid loading the dependency altogether when only parser is called to generate the manual page but the build server does not have all the dependencies required by the specific addon module.

https://trac.osgeo.org/grass/search?q=lazy+import&noquickjump=1&changeset=on

An idea is to try make use of importlib.import_module() along with
sys, like:

import sys
import importlib
if 'xyz' not in sys.modules:
importlib.import_module('xyz')

or better (?):

if 'xyz' not in sys.modules and 'xyz' not in dir():
importlib.import_module('xyz')

Nikos, this is a valid approach for import magics but an overkill for this. Python import statement can be anywhere and is sufficient for most cases. importlib is for things like dynamic loading (e.g. plugins to GUI). Martin’s answer:

https://lists.osgeo.org/pipermail/grass-dev/2018-October/090321.html

5

Hi,

pá 19. 10. 2018 v 17:18 odesílatel Martin Landa <landa.martin@gmail.com> napsal:

pá 19. 10. 2018 v 16:45 odesílatel Markus Neteler <neteler@osgeo.org> napsal:

we are currently writing a Python script which needs to import some
heavy external libraries.

if I understand well, a lazy import could help

“”"
def main:
from mylib import xyz
…

return 0

if name == “main”:
opt, flg = grass.parser()
sys.exit(main())
“”"

In my AddOn [1], I followed exactly this approach to avoid double import of TensorFlow.

If I remember correctly, libraries imported before the call of grass.parser() are imported also during the installation through g.extension because of checks or something and you really don’t want to import quite heavy TensorFlow (which is moreover printing some warnings during the import on most of computers) making your installation slow. The same applied to the double import when running the script.

[1] https://trac.osgeo.org/grass/browser/grass-addons/grass7/imagery/i.ann.maskrcnn/i.ann.maskrcnn.train/i.ann.maskrcnn.train.py?rev=72730#L181

6

Hi,

On Fri, Oct 19, 2018 at 5:18 PM Martin Landa <landa.martin@gmail.com> wrote:

pá 19. 10. 2018 v 16:45 odesílatel Markus Neteler <neteler@osgeo.org> napsal:
> we are currently writing a Python script which needs to import some
> heavy external libraries.

if I understand well, a lazy import could help

"""
def main:
    from mylib import xyz
    ...

   return 0

if __name__ == "__main__":
    opt, flg = grass.parser()
    sys.exit(main())
"""

yes, this approach works nicely.

Thanks!
Markus