[GRASS-dev] Calling GRASS from outside: cannot open shared object file

wenzeslaus · February 18, 2013, 5:41pm

Hi,

I've tried to use GRASS without starting it explicitly in Python. I've
used example from wiki [1] which works. However, when I tried to use
library functions through ctypes, I'm not able to import lib modules.
I've added these lines to the example:

import grass.lib.gis as gis
print gis.G_location_path()

and the import fails with and error:

Traceback (most recent call last):
  File "python_test_ctypes.py", line 23, in <module>
    import grass.lib.gis as gis
  File "/home/vasek/dev/grass/trunk/dist.i686-pc-linux-gnu/etc/python/grass/lib/gis.py",
line 23, in <module>
    _libs["grass_gis.7.0.svn"] = load_library("grass_gis.7.0.svn")
  File "/home/vasek/dev/grass/trunk/dist.i686-pc-linux-gnu/etc/python/grass/lib/ctypes_loader.py",
line 55, in load_library
    return self.load(path)
  File "/home/vasek/dev/grass/trunk/dist.i686-pc-linux-gnu/etc/python/grass/lib/ctypes_loader.py",
line 72, in load
    raise ImportError,e
ImportError: libgrass_datetime.7.0.svn.so: cannot open shared object
file: No such file or directory

I placed print path to the ctypes/loader.py file (line 71) and I get
this output:

/home/vasek/dev/grass/trunk/dist.i686-pc-linux-gnu/lib/libgrass_gis.7.0.svn.so

This file exists (checked on disk and also python code checks this in
the function load_library) and libgrass_datetime.7.0.svn.so exists in
the same folder, too. The output of print
os.environ['LD_LIBRARY_PATH'] is:

:/home/vasek/dev/grass/trunk/dist.i686-pc-linux-gnu/lib

Running this script in GRASS session works.

Vaclav

[1] http://grasswiki.osgeo.org/wiki/Working_with_GRASS_without_starting_it_explicitly#Python_example
[2] http://grasswiki.osgeo.org/wiki/GRASS_and_Python#Creating_Python_scripts_that_call_GRASS_functionality_from_outside
(BTW, these to sections could be merged together.)

(attachments)

python_test_ctypes.py (556 Bytes)

wenzeslaus · May 9, 2014, 3:55am

Hi Markus,

going back to list because it is apparently a general problem. The answer is no. I haven’t resolved this.

Intro for others: currently, it is not possible to use ctypes API (and thus PyGRASS) in standalone Python script which is not GRASS module (script running outside GRASS session).

The error is:

…lib/ctypes_loader.py", line 71, in load
raise ImportError,e
ImportError: libgrass_datetime.7.1.svn.so: cannot open shared object file: No such file or directory

On Linux print os.environ['LD_LIBRARY_PATH'] shows the path to GRASS libs (:/.../dist.../lib). And explicit setting of this variable (without leading :) does not change anything.

It seems that it is not possible to influence library loading by LD_LIBRARY_PATH when the process is already running. [1] And ctypes does not seem to have special support to handle path adding at runtime. [2, 3]

Is this something library loader in GRASS should take care of?

Vaclav

[1] http://stackoverflow.com/questions/856116/changing-ld-library-path-at-runtime-for-ctypes
[2] https://docs.python.org/2/library/ctypes.html
[3] http://bugs.python.org/issue2936

···

On Thu, May 8, 2014 at 5:09 PM, Markus Neteler <neteler@osgeo.org> wrote:

Hi Vaclav,

did you ever resolve this issue? I have it now (ok, with some delay to
your tests :-).
Attached my script which is yours with a few minor additions.

Would be glad to learn how to fix this issue:

python ~/python_test_ctypes.py
{‘MAPSET’: ‘user1’, ‘GISDBASE’: ‘/home/neteler/grassdata’,
‘LOCATION_NAME’: ‘nc_spm_08_grass7’}
Raster maps:
lsat5_1987_10@landsat
lsat5_1987_40@landsat
…
zero@user1
horangle_01@user1
precip_annual@user1

Traceback (most recent call last):

File “/home/neteler/python_test_ctypes.py”, line 38, in
import grass.lib.gis as gis
File “/home/neteler/software/grass71/dist.x86_64-unknown-linux-gnu/etc/python/grass/lib/gis.py”,
line 23, in
_libs[“grass_gis.7.1.svn”] = load_library(“grass_gis.7.1.svn”)
File “/home/neteler/software/grass71/dist.x86_64-unknown-linux-gnu/etc/python/grass/lib/ctypes_loader.py”,

line 55, in load_library
return self.load(path)

File “/home/neteler/software/grass71/dist.x86_64-unknown-linux-gnu/etc/python/grass/lib/ctypes_loader.py”,
line 71, in load
raise ImportError,e
ImportError: libgrass_datetime.7.1.svn.so: cannot open shared object

file: No such file or directory

[neteler@oboe python_intro]$

thanks
Markus

On Mon, Feb 18, 2013 at 6:41 PM, Vaclav Petras <wenzeslaus@gmail.com> wrote:

Hi,

I’ve tried to use GRASS without starting it explicitly in Python. I’ve
used example from wiki [1] which works. However, when I tried to use
library functions through ctypes, I’m not able to import lib modules.
I’ve added these lines to the example:

import grass.lib.gis as gis
print gis.G_location_path()

and the import fails with and error:

Traceback (most recent call last):
File “python_test_ctypes.py”, line 23, in
import grass.lib.gis as gis
File “/home/vasek/dev/grass/trunk/dist.i686-pc-linux-gnu/etc/python/grass/lib/gis.py”,
line 23, in
_libs[“grass_gis.7.0.svn”] = load_library(“grass_gis.7.0.svn”)
File “/home/vasek/dev/grass/trunk/dist.i686-pc-linux-gnu/etc/python/grass/lib/ctypes_loader.py”,
line 55, in load_library
return self.load(path)
File “/home/vasek/dev/grass/trunk/dist.i686-pc-linux-gnu/etc/python/grass/lib/ctypes_loader.py”,
line 72, in load
raise ImportError,e
ImportError: libgrass_datetime.7.0.svn.so: cannot open shared object
file: No such file or directory

I placed print path to the ctypes/loader.py file (line 71) and I get
this output:

/home/vasek/dev/grass/trunk/dist.i686-pc-linux-gnu/lib/libgrass_gis.7.0.svn.so

This file exists (checked on disk and also python code checks this in
the function load_library) and libgrass_datetime.7.0.svn.so exists in
the same folder, too. The output of print
os.environ[‘LD_LIBRARY_PATH’] is:

:/home/vasek/dev/grass/trunk/dist.i686-pc-linux-gnu/lib

Running this script in GRASS session works.

Vaclav

[1] http://grasswiki.osgeo.org/wiki/Working_with_GRASS_without_starting_it_explicitly#Python_example
[2] http://grasswiki.osgeo.org/wiki/GRASS_and_Python#Creating_Python_scripts_that_call_GRASS_functionality_from_outside
(BTW, these to sections could be merged together.)

grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

neteler · May 9, 2014, 8:59am

Hi,

I get it running (on Linux, winGRASS untested) when setting up the
LD_LIBRARY_PATH at system level:

su
echo "`grass71 --config path`/lib" > /etc/ld.so.conf.d/grass71.conf
ldconfig
exit

# run as normal user:
python python_test_ctypes.py
...
precip_annual@user1
/home/neteler/grassdata/nc_spm_08_grass7

Works.
The issue is that we cannot easily enforce the users to add a file in
/etc/ld.so.conf.d/ since it requires admin rights.

Suggestions?

Markus

(attachments)

python_test_ctypes.py (1.14 KB)

mlennert · May 9, 2014, 9:52am

On 09/05/14 10:59, Markus Neteler wrote:

Hi,

I get it running (on Linux, winGRASS untested) when setting up the
LD_LIBRARY_PATH at system level:

su
echo "`grass71 --config path`/lib" > /etc/ld.so.conf.d/grass71.conf
ldconfig
exit

# run as normal user:
python python_test_ctypes.py
...
precip_annual@user1
/home/neteler/grassdata/nc_spm_08_grass7

Works.
The issue is that we cannot easily enforce the users to add a file in
/etc/ld.so.conf.d/ since it requires admin rights.

Suggestions?

Doesn't this work:

export LD_LIBRARY_PATH= export LD_LIBRARY_PATH=$(grass71 --config path)/lib
python python_test_ctypes.py

?

Moritz--
Département de Géographie
Université Libre de Bruxelles
CP 130/03
Av. F.D. Roosevelt 50
1050 Bruxelles
Belgique

tél. + 32 2 650.68.12 / 68.11 (secr.)
fax + 32 2 650.68.30

neteler · May 9, 2014, 1:12pm

On Fri, May 9, 2014 at 11:52 AM, Moritz Lennert
<mlennert@club.worldonline.be> wrote:

On 09/05/14 10:59, Markus Neteler wrote:

Hi,

I get it running (on Linux, winGRASS untested) when setting up the
LD_LIBRARY_PATH at system level:

su
echo "`grass71 --config path`/lib" > /etc/ld.so.conf.d/grass71.conf
ldconfig
exit

# run as normal user:
python python_test_ctypes.py
...
precip_annual@user1
/home/neteler/grassdata/nc_spm_08_grass7

Works.
The issue is that we cannot easily enforce the users to add a file in
/etc/ld.so.conf.d/ since it requires admin rights.

Suggestions?

Doesn't this work:

export LD_LIBRARY_PATH= export LD_LIBRARY_PATH=$(grass71 --config path)/lib
python python_test_ctypes.py

Yes this works:

export LD_LIBRARY_PATH=$(grass71 --config path)/lib
python python_test_ctypes.py

also this:
export LD_LIBRARY_PATH=`grass71 --config path`/lib
python python_test_ctypes.py

nice, I could have tried that earlier

Now we just need similar instructions for Windows and MacOSX.

Markus

wenzeslaus · May 9, 2014, 1:31pm

On Fri, May 9, 2014 at 9:12 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Fri, May 9, 2014 at 11:52 AM, Moritz Lennert
<mlennert@club.worldonline.be> wrote:
> On 09/05/14 10:59, Markus Neteler wrote:
>>
>> Hi,
>>
>> I get it running (on Linux, winGRASS untested) when setting up the
>> LD_LIBRARY_PATH at system level:
>>
>> su
>> echo "`grass71 --config path`/lib" > /etc/ld.so.conf.d/grass71.conf
>> ldconfig
>> exit
>>
>> # run as normal user:
>> python python_test_ctypes.py
>> ...
>> precip_annual@user1
>> /home/neteler/grassdata/nc_spm_08_grass7
>>
>> Works.
>> The issue is that we cannot easily enforce the users to add a file in
>> /etc/ld.so.conf.d/ since it requires admin rights.
>>
>> Suggestions?
>
>
> Doesn't this work:
>
> export LD_LIBRARY_PATH= export LD_LIBRARY_PATH=$(grass71 --config
path)/lib
> python python_test_ctypes.py

Yes this works:

export LD_LIBRARY_PATH=$(grass71 --config path)/lib
python python_test_ctypes.py

also this:
export LD_LIBRARY_PATH=`grass71 --config path`/lib
python python_test_ctypes.py

nice, I could have tried that earlier

I'm not sure if this is something I want to do each time before starting

the script, especially when I'm setting all the paths inside the script
during imports and then I call `grass.script.setup.init` function.

Now we just need similar instructions for Windows and MacOSX.

Again, this is only a workaround. I don't think that we want to explain
working with shared/dynamic libraries on all platforms to all scripting
GRASS users.

I would like to know if it is possible to manage this inside `grass.lib` or
through `grass.script.setup.init`.

Vaclav

Glynn_Clements1 · May 10, 2014, 5:31pm

Vaclav Petras wrote:

going back to list because it is apparently a general problem. The answer
is no. I haven't resolved this.

Intro for others: currently, it is not possible to use ctypes API (and thus
PyGRASS) in standalone Python script which is not GRASS module (script
running outside GRASS session).

Even if you can load the library, you probably aren't going to be able
to do much with it unless GISRC is set.

--
Glynn Clements <glynn@gclements.plus.com>

wenzeslaus · May 10, 2014, 5:38pm

On Sat, May 10, 2014 at 1:31 PM, Glynn Clements <glynn@gclements.plus.com>wrote:

Vaclav Petras wrote:

> going back to list because it is apparently a general problem. The answer
> is no. I haven't resolved this.
>
> Intro for others: currently, it is not possible to use ctypes API (and
thus
> PyGRASS) in standalone Python script which is not GRASS module (script
> running outside GRASS session).

Even if you can load the library, you probably aren't going to be able
to do much with it unless GISRC is set.

I hoped that this is what `grass.script.setup.init` is doing.

--

Glynn Clements <glynn@gclements.plus.com>

Glynn_Clements1 · May 12, 2014, 10:23pm

Vaclav Petras wrote:

> Even if you can load the library, you probably aren't going to be able
> to do much with it unless GISRC is set.

I hoped that this is what `grass.script.setup.init` is doing.

Fair enough.

However, that's going to be subject to a number of of chicken-and-egg
problems, of which LD_LIBRARY_PATH is only one (e.g. on Windows with a
"bundled" Python interpreter, running a Python script from outside
GRASS is going to use the system Python, not the bundled Python).

The safe solution is to first set up all of the environment variables
for a GRASS "session" then execute the "real" script with e.g.
os.execv().

That should eliminate any issues with changes to environment variables
not taking effect. It should also ensure that the real script is
executed using any bundled Python.

To be honest, I'm not sure that setup.py belongs in lib/python.
lib/init (alongside grass.py) seems more appropriate, as it's widely
understood that any code in that directory can't rely upon the usual
GRASS features.

--
Glynn Clements <glynn@gclements.plus.com>

neteler · May 13, 2014, 4:36pm

Hi,

I tried my current start-python-and-create-a-location script on Windows. The location is generated but then a call of the scripting library fails with a gettext error (I print the os.environment to understand more but no idea):

Python 2.7.6 (default, Nov 10 2013, 19:24:18) [MSC v.1500 32 bit (Intel)] on win32
Type “help”, “copyright”, “credits” or “license” for more information.

Imported NumPy 1.8.0, SciPy 0.13.3, Matplotlib 1.3.1
runfile(‘C:/Users/VPCF2/Desktop/STEM/grass7_python/start_grass7_create_new_location_ADVANCED (1).py’, wdir=r’C:/Users/VPCF2/Desktop/STEM/grass7_python’)

guidata 1.6.1, guiqwt 2.3.1
Type “scientific” for more details.

Created location c:\users\vpcf2\appdata\local\temp\grassdata\389cf64663f4d727d9ca9864e2e6de64
{‘TMP’: ‘C:\Users\VPCF2\AppData\Local\Temp’, ‘COMPUTERNAME’: ‘VPCF2-VAIO’, ‘SPYDER_PARENT_DIR’: ‘C:\Python27\lib\site-packages’, ‘ETS_TOOLKIT’: ‘qt4’, ‘USERDOMAIN’: ‘VPCF2-VAIO’, ‘QT_API’: ‘pyqt’, ‘EMC_AUTOPLAY’: ‘C:\Program Files (x86)\Common Files\Roxio Shared\’, ‘PSMODULEPATH’: ‘C:\Windows\system32\WindowsPowerShell\v1.0\Modules\’, ‘COMMONPROGRAMFILES’: ‘C:\Program Files (x86)\Common Files’, ‘PROCESSOR_IDENTIFIER’: ‘Intel64 Family 6 Model 42 Stepping 7, GenuineIntel’, ‘IGNORE_SIP_SETAPI_ERRORS’: ‘False’, ‘PROGRAMFILES’: ‘C:\Program Files (x86)’, ‘PROCESSOR_REVISION’: ‘2a07’, ‘SYSTEMROOT’: ‘C:\Windows’, ‘PATH’: ‘C:\Python27\Lib\site-packages\PyQt4;C:\Program Files\ImageMagick-6.8.6-Q16;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Common Files\Microsoft Shared\Windows Live;C:\Program Files (x86)\Common Files\Microsoft Shared\Windows Live;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\WIDCOMM\Bluetooth Software\;C:\Program Files\WIDCOMM\Bluetooth Software\syswow64;\;C:\Program Files (x86)\Sony\VAIO Startup Setting Tool;C:\Program Files (x86)\Common Files\Roxio Shared\10.0\DLLShared\;C:\Program Files (x86)\Common Files\Roxio Shared\DLLShared\;C:\Program Files (x86)\Windows Live\Shared;C:\Python27;C:\Python27\DLLs;C:\Python27\Scripts;C:\Python27\gnuplot\binary;C:\Program Files (x86)\pythonxy\SciTE-3.3.2-3;C:\Program Files (x86)\pythonxy\console;C:\MinGW32-xy\bin’, ‘ROXIOCENTRAL’: ‘C:\Program Files (x86)\Common Files\Roxio Shared\10.0\Roxio Central36\’, ‘PROGRAMFILES(X86)’: ‘C:\Program Files (x86)’, ‘COMSPEC’: ‘C:\Windows\system32\cmd.exe’, ‘LANG’: ‘en_US’, ‘TK_LIBRARY’: ‘C:\Python27\tcl\tk8.5’, ‘CONFIGSETROOT’: ‘C:\Windows\ConfigSetRoot’, ‘UMD_NAMELIST’: ‘guidata,guiqwt’, ‘TEMP’: ‘C:\Users\VPCF2\AppData\Local\Temp’, ‘UMD_VERBOSE’: ‘True’, ‘PROCESSOR_ARCHITECTURE’: ‘x86’, ‘LD_LIBRARY_PATH’: ‘C:\OSGeo4W\apps\grass\grass-7.1.svn\lib’, ‘TIX_LIBRARY’: ‘C:\Python27\tcl\tix8.4.3’, ‘ALLUSERSPROFILE’: ‘C:\ProgramData’, ‘LOCALE’: ‘C’, ‘MATPLOTLIB_BACKEND’: ‘Qt4Agg’, ‘SESSIONNAME’: ‘Console’, ‘HOMEPATH’: ‘\Users\VPCF2’, ‘PROGRAMW6432’: ‘C:\Program Files’, ‘USERNAME’: ‘VPCF2’, ‘SPYDER_AR_STATE’: ‘True’, ‘LOGONSERVER’: ‘\\VPCF2-VAIO’, ‘LOCALAPPDATA’: ‘C:\Users\VPCF2\AppData\Local’, ‘PROGRAMDATA’: ‘C:\ProgramData’, ‘PYTHONPATH’: ‘C:\Users\VPCF2\.qgis2\processing;C:\OSGeo4W\apps\grass\grass-7.1.svn\scripts’, ‘SPYDER_N_PORT’: ‘20129’, ‘SPYDER_SHELL_ID’: ‘146823976’, ‘TCL_LIBRARY’: ‘C:\Python27\tcl\tcl8.5’, ‘PUBLIC’: ‘C:\Users\Public’, ‘PATHEXT’: ‘.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW’, ‘UMD_ENABLED’: ‘True’, ‘MATPLOTLIB_PATCH’: ‘None’, ‘FP_NO_HOST_CHECK’: ‘NO’, ‘WINDIR’: ‘C:\Windows’, ‘GISDBASE’: ‘c:\users\vpcf2\appdata\local\temp\grassdata’, ‘IPYTHON_KERNEL’: ‘False’, ‘SPYDER_AR_TIMEOUT’: ‘2000’, ‘HOMEDRIVE’: ‘C:’, ‘INSTALL_QT_INPUTHOOK’: ‘True’, ‘PYTHONSTARTUP’: ‘C:\Python27\lib\site-packages\spyderlib\scientific_startup.py’, ‘SYSTEMDRIVE’: ‘C:’, ‘COLORIZE_SYS_STDERR’: ‘False’, ‘NUMBER_OF_PROCESSORS’: ‘8’, ‘APPDATA’: ‘C:\Users\VPCF2\AppData\Roaming’, ‘SPYDER_I_PORT’: ‘20128’, ‘PROCESSOR_LEVEL’: ‘6’, ‘SPYDER_ENCODING’: ‘cp1252’, ‘PROCESSOR_ARCHITEW6432’: ‘AMD64’, ‘GRASS_SH’: ‘C:\OSGeo4W\apps\grass\grass-7.1.svn\msys\bin\sh.exe’, ‘COMMONPROGRAMW6432’: ‘C:\Program Files\Common Files’, ‘OS’: ‘Windows_NT’, ‘COMMONPROGRAMFILES(X86)’: ‘C:\Program Files (x86)\Common Files’, ‘USERPROFILE’: ‘C:\Users\VPCF2’}
Traceback (most recent call last):
File “”, line 1, in
File “C:\Python27\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py”, line 540, in runfile
execfile(filename, namespace)
File “C:/Users/VPCF2/Desktop/STEM/grass7_python/start_grass7_create_new_location_ADVANCED (1).py”, line 127, in
import grass.script as grass
File “C:\OSGeo4W\apps\grass\grass-7.1.svn\etc\python\grass\script_init_.py”, line 1, in
from core import *
File “C:\OSGeo4W\apps\grass\grass-7.1.svn\etc\python\grass\script\core.py”, line 38, in
gettext.install(‘grasslibs’, os.path.join(os.getenv(“GISBASE”), ‘locale’))
File “C:\Python27\lib\ntpath.py”, line 96, in join
assert len(path) > 0
TypeError: object of type ‘NoneType’ has no len()

Any suggestions? This is using OSGeo4W 32bit.

thanks
Markus

Glynn_Clements1 · May 13, 2014, 8:23pm

Markus Neteler wrote:

I tried my current start-python-and-create-a-location script on Windows.
The location is generated but then a call of the scripting library fails
with a gettext error (I print the os.environment to understand more but no
idea):

"C:\OSGeo4W\apps\grass\grass-7.1.svn\etc\python\grass\script\core.py", line 38, in <module>
    gettext.install('grasslibs', os.path.join(os.getenv("GISBASE"), 'locale'))
  File "C:\Python27\lib\ntpath.py", line 96, in join
    assert len(path) > 0
TypeError: object of type 'NoneType' has no len()

Your dump of os.environ doesn't mention GISBASE.

os.getenv("GISBASE") will return None, and pass it to os.path.join(),
where the "assert len(path) > 0" (line 96 of ntpath.py) will call
len(None), which results in the above TypeError exception.

--
Glynn Clements <glynn@gclements.plus.com>

neteler · May 14, 2014, 12:06pm

On Tue, May 13, 2014 at 10:23 PM, Glynn Clements
<glynn@gclements.plus.com> wrote:

Markus Neteler wrote:

I tried my current start-python-and-create-a-location script on Windows.
The location is generated but then a call of the scripting library fails
with a gettext error (I print the os.environment to understand more but no
idea):

"C:\OSGeo4W\apps\grass\grass-7.1.svn\etc\python\grass\script\core.py", line 38, in <module>
    gettext.install('grasslibs', os.path.join(os.getenv("GISBASE"), 'locale'))
  File "C:\Python27\lib\ntpath.py", line 96, in join
    assert len(path) > 0
TypeError: object of type 'NoneType' has no len()

Your dump of os.environ doesn't mention GISBASE.

os.getenv("GISBASE") will return None, and pass it to os.path.join(),
where the "assert len(path) > 0" (line 96 of ntpath.py) will call
len(None), which results in the above TypeError exception.

Right, forgot to set it for Windows (in the Linux subsection it was there).
Now it works.

thanks
Markus