[GRASS-dev] [GRASS GIS] #2424: PyGRASS does not work when GRASS is invoked from outside

#2424: PyGRASS does not work when GRASS is invoked from outside
------------------------------------------------------+---------------------
Reporter: wenzeslaus | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python ctypes | Version: svn-trunk
Keywords: installation, pygrass, temporal, scripts | Platform: MSWindows 8
      Cpu: Unspecified |
------------------------------------------------------+---------------------
When one want to
[http://grasswiki.osgeo.org/wiki/Working_with_GRASS_without_starting_it_explicitly
work with GRASS without starting it explicitly], it is possible to use
`grass.script` but it is not possible to use `grass.pygrass`,
`grass.temporal` and `grass.lib`.

This might have been discussed on mailing already, anyway the reason is
that the dynamic libraries are not accessible to the current process.

Output on Linux:

{{{
Traceback (most recent call last):
   File "./run_grass_from_outside.py", line 62, in <module>
     from grass.pygrass import vector
   File "/opt/src/grass-trunk/dist.x86_64-unknown-linux-
gnu/etc/python/grass/pygrass/__init__.py", line 7, in <module>
     import grass.lib.gis as _libgis
   File "/opt/src/grass-trunk/dist.x86_64-unknown-linux-
gnu/etc/python/grass/lib/gis.py", line 23, in <module>
     _libs["grass_gis.7.1.svn"] = load_library("grass_gis.7.1.svn")
   File "/opt/src/grass-trunk/dist.x86_64-unknown-linux-
gnu/etc/python/grass/lib/ctypes_loader.py", line 55, in load_library
     return self.load(path)
   File "/opt/src/grass-trunk/dist.x86_64-unknown-linux-
gnu/etc/python/grass/lib/ctypes_loader.py", line 71, in load
     raise ImportError,e
ImportError: libgrass_datetime.7.1.svn.so: cannot open shared object file:
No such file or directory
}}}

Output on MS Windows:

{{{
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Anaconda\lib\site-
packages\spyderlib\widgets\externalshell\sitecustomize.py", line 523, in
runfile
    execfile(filename, namespace)
  File "C:\Users\akratoc\test_grass_standalone.py", line 59, in <module>
    from grass.pygrass import vector
  File "C:\Program Files (x86)\GRASS GIS
7.0.0beta3\etc\python\grass\pygrass\__init__.py", line 7, in <module>
    import grass.lib.gis as _libgis
  File "C:\Program Files (x86)\GRASS GIS
7.0.0beta3\etc\python\grass\lib\gis.py", line 23, in <module>
    _libs["grass_gis.7.0.0beta3"] = load_library("grass_gis.7.0.0beta3")
  File "C:\Program Files (x86)\GRASS GIS
7.0.0beta3\etc\python\grass\lib\ctypes_loader.py", line 55, in
load_library
    return self.load(path)
  File "C:\Program Files (x86)\GRASS GIS
7.0.0beta3\etc\python\grass\lib\ctypes_loader.py", line 221, in load
    return _WindowsLibrary(path)
  File "C:\Program Files (x86)\GRASS GIS
7.0.0beta3\etc\python\grass\lib\ctypes_loader.py", line 207, in _init_
    self.cdll = ctypes.cdll.LoadLibrary(path)
  File "C:\Anaconda\lib\ctypes\__init__.py", line 443, in LoadLibrary
    return self._dlltype(name)
  File "C:\Anaconda\lib\ctypes\__init__.py", line 365, in _init_
    self._handle = _dlopen(self._name, mode)
WindowsError: [Error 193] %1 is not a valid Win32 application
}}}

It is unfortunate that you even cannot import `grass.pygrass.modules`:

{{{
from grass.pygrass import modules
}}}

For Linux, we have to rely on decision of packagers and a user being able
to manage `~/.bashrc` or environmental variables of the given process.

For MS Windows, the solution would be to allow user to add the variables
to the environment. It seems to me that it is "enough" to add path with
dynamic libraries (`lib` and `extrabin`) and GRASS installation directory
(the one with the `grass*.bat` file) to `PATH` and Python packages
(`etc/python`) to `PYTHONPATH`.

The solution is of course invasive but there is no other way. A lot of
applications just do the same without worrying about consequences. We can
try what Git for MS Windows is doing: giving a choice how git commands and
other tools should be spread into the system (modify or not modify the
`PATH`).

The question of course is what should be the default.

Other question is whether to add to the beginning or the end of `PATH` and
`PYTHONPATH` variables.

Also, there must be a warning (in MS Windows installer) that no other
version of GRASS GIS will work.

The same might apply to other OSGeo or related software. This is
unfortunately unknown. I guess that this option cannot be available in
OSGeo4W installer.

I have no idea about Mac OS X. Which solution would be appropriate there?

Do you have some other ideas how to solve or workaround this issue? Would
it be possible to provide a script/functionality in GRASS GIS which would
put the things to environmental variables and take it away if requested (I
guess that the removal is necessary for standard installation too)? This
could be much more flexible. This might be even applicable to all
platforms although for MS Windows it would be quite different from Linux.

Related:
  * #2333 (choose python interpreter during the GRASS installation on
windows)

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2424&gt;
GRASS GIS <http://grass.osgeo.org>

#2424: PyGRASS does not work when GRASS is invoked from outside
------------------------------------------------------+---------------------
Reporter: wenzeslaus | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python ctypes | Version: svn-trunk
Keywords: installation, pygrass, temporal, scripts | Platform: MSWindows 8
      Cpu: Unspecified |
------------------------------------------------------+---------------------

Comment(by neteler):

On Linux, I had success with this startup:

{{{
export LD_LIBRARY_PATH=$(grass70 --config path)/lib
python start_grass7_create_new_location_ADVANCED.py
}}}

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2424#comment:1&gt;
GRASS GIS <http://grass.osgeo.org>

#2424: PyGRASS does not work when GRASS is invoked from outside
------------------------------------------------------+---------------------
Reporter: wenzeslaus | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python ctypes | Version: svn-trunk
Keywords: installation, pygrass, temporal, scripts | Platform: MSWindows 8
      Cpu: Unspecified |
------------------------------------------------------+---------------------

Comment(by wenzeslaus):

Replying to [comment:1 neteler]:
> On Linux, I had success with this startup:
>
{{{
export LD_LIBRARY_PATH=$(grass70 --config path)/lib
python start_grass7_create_new_location_ADVANCED.py
}}}

Sure, that's the problem. For Linux user it is incredibly simple, for MS
Windows one, incredibly hard.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2424#comment:2&gt;
GRASS GIS <http://grass.osgeo.org>

#2424: PyGRASS does not work when GRASS is invoked from outside
------------------------------------------------------+---------------------
Reporter: wenzeslaus | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python ctypes | Version: svn-trunk
Keywords: installation, pygrass, temporal, scripts | Platform: MSWindows 8
      Cpu: Unspecified |
------------------------------------------------------+---------------------

Comment(by wenzeslaus):

Another case when I'm hitting this issue is in running tests. There is a
script which runs the tests. I import form `gunittest` in this test and I
don't really need much but in `__init__.py` I import some things to
provide and easy access, so whatever I want to import,
[source:grass/trunk/lib/python/gunittest/__init__.py __init__.py] gets
executed, because `pygrass.modules` are used this executes the `pygrass`
main [source:grass/trunk/lib/python/pygrass/__init__.py __init__.py] which
imports some PyGRASS things which depend on `ctypes` interface and when
dynamic libraries are not available, this obviously fails. This is an
example, how simple you may need dynamic libraries. Temporal framework
Python API is probably the same case.

In some cases you can get around this by putting try excepts to the
library `__init__.py`, as I did for
[source:grass/trunk/lib/python/gunittest/__init__.py?rev=61489#L20
gunittest].

Another way how to improve the situation is to use empty `__init__.py`
files. We are not doing it now, we are doing almost the opposite and it of
course has a big advantage as hiding the inner structure of the package.
On the other hand, from `grass.script` or `grass.pygrass` you usually need
just a specific part. There is no consensus but there is a lot of people
thinking that empty `__init__.py` files are the best. You anyway don't
gain much by hiding the inner structure of the package unless you forbid
the specific imports in your API doc and we are not doing this. This is of
course something we should sort out before the release (sorry about that).

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2424#comment:3&gt;
GRASS GIS <http://grass.osgeo.org>

#2424: PyGRASS does not work when GRASS is invoked from outside
------------------------------------------------------+---------------------
Reporter: wenzeslaus | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python ctypes | Version: svn-trunk
Keywords: installation, pygrass, temporal, scripts | Platform: MSWindows 8
      Cpu: Unspecified |
------------------------------------------------------+---------------------

Comment(by zarch):

Replying to [comment:2 wenzeslaus]:
> Another case when I'm hitting this issue is in running tests.
> There is a script which runs the tests. I import form gunittest
> in this test and I don't really need much but in __init__.py
> I import some things to provide and easy access, so whatever
> I want to import, __init__.py gets executed, because pygrass.modules
> are used this executes the pygrass main __init__.py which imports
> some PyGRASS things which depend on ctypes interface and when
> dynamic libraries are not available, this obviously fails.
> This is an example, how simple you may need dynamic libraries.
> Temporal framework Python API is probably the same case.

Since the pygrass.modules does not depend from the ctypes, I think we
could move the modules stuff outside pygrass to remove the depedencies to
ctypes...

The main problem is that I put in pygrass.__init__:

{{{
import grass.lib.gis as _libgis
_libgis.G_gisinit('')
}}}

Because all the pygrass sub-modules: gis, raster, vector and functions
must have G_gisinit,
If you think that move the pygrass.modules outside pyrass is not a good
idea, I can move the G_gisinit from the main __init__ of pygrass to the
sub-modules.

> Another way how to improve the situation is to use
> empty __init__.py files.

Generally I'm not against to an empty or almost empty init file.
So If you agree on that, I could change it.

This it will change the way as we import stuff, so it wont be possible to
import:

{{{
from grass import pygrass

# and then access to the sub-modules
pygrass.modules.Module
}}}

For me it is not a problem, generally I import the class that I need on
the top, like:

{{{
from grass.pygrass.modules import Module
}}}

In any case I think that the pygrass.modules.__init__ should contain at
least:

{{{
from grass.pygrass.modules.interface.module import Module,
ParallelModuleQueue
}}}

just to make the use of these classes more convenient and to "hide" the
internal structure:

{{{
modules/
   interface/
     module.py
     ...
}}}

Otherwise it became too verbose.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2424#comment:4&gt;
GRASS GIS <http://grass.osgeo.org>

#2424: PyGRASS does not work when GRASS is invoked from outside
------------------------------------------------------+---------------------
Reporter: wenzeslaus | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python ctypes | Version: svn-trunk
Keywords: installation, pygrass, temporal, scripts | Platform: MSWindows 8
      Cpu: Unspecified |
------------------------------------------------------+---------------------

Comment(by wenzeslaus):

Replying to [comment:4 zarch]:
>
> Since the pygrass.modules does not depend from the ctypes, I think we
could move the modules stuff outside pygrass to remove the depedencies to
ctypes...
>
> The main problem is that I put in `pygrass.__init__`:
>
{{{
import grass.lib.gis as _libgis
_libgis.G_gisinit('')
}}}
>
> Because all the pygrass sub-modules: gis, raster, vector and functions
must have `G_gisinit`,
> If you think that move the pygrass.modules outside pyrass is not a good
idea, I can move the `G_gisinit` from the main `__init__` of pygrass to
the sub-modules.
>
I cannot comment on `G_gisinit` much but the error is that the DLL is not
found, so I though that it is before the call.
>
> > Another way how to improve the situation is to use
> > empty `__init__.py` files.
>
> Generally I'm not against to an empty or almost empty init file.
> So If you agree on that, I could change it.
>
> This it will change the way as we import stuff, so it wont be possible
to import:
>
{{{
from grass import pygrass

# and then access to the sub-modules
pygrass.modules.Module
}}}
>
> For me it is not a problem, generally I import the class that I need on
the top, like:
>
{{{
from grass.pygrass.modules import Module
}}}
>
This is what I do too, the same for `grass.script`. But `grass.script` is
an example where some things were used for compatibility and changes in
internal structure are expected in the future I would say. But generally,
yes, it seems that it is better to have empty `__init__.py` files.

I'm not sure if it is better to move `pygrass.modules` outside of
`pygrass` or not. I was not thinking about it much. However, it seems that
`pygrass.modules` is really something different from the library
interface. The only connection is that it is new and Pythonic and it was
implemented together. The `ctypes` dependency is quite a big difference.
I'm not sure if there will be something in the future which would
introduce `ctypes` to modules. Perhaps yes, for example some command line
parsing interface using GRASS parser (based on library rather then on
`g.parser`). What would be the name of a standalone package? Just
`grass.modules`?

> In any case I think that the `pygrass.modules.__init__` should contain
at least:
>
{{{
from grass.pygrass.modules.interface.module import Module,
ParallelModuleQueue
}}}
>
> just to make the use of these classes more convenient and to "hide" the
internal structure:
>
> Otherwise it became too verbose.

I agree, this is really internal and `__init__.py` should be used to hide
it.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2424#comment:5&gt;
GRASS GIS <http://grass.osgeo.org>