[GRASS-dev] [GRASS GIS] #1646: GRASS ctypes exception handling

#1646: GRASS ctypes exception handling
-----------------------------------------------------+----------------------
Reporter: huhabla | Owner: grass-dev@…
     Type: enhancement | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python ctypes | Version: svn-trunk
Keywords: setjmp, longjmp, Exception, gently exit | Platform: All
      Cpu: All |
-----------------------------------------------------+----------------------
This enhancement request is based on the GRASS GIS developers mailing list
discussion in [1].

The basic idea is to catch fatal error calls in Python when using the
ctypes GRASS library wrapper. Catching the exit call in case of a fatal
error is needed to gently exit the calling Python module. So open file
descriptors or database connections can be closed safely, unfinished
imports/exports or temporary data can be removed correctly,
region/mapset/location state of the current session can be reconstructed
(calling GRASS modules not library functions).

Glynn suggested a solution to this question:

Soeren: ''Is it possible to raise a Python exception instead of calling
exit in
case of a fatal error when using ctypes wrapped GRASS library functions?''

Glynn: ''Yes, but you would have to wrap each function individually.''

Glynn suggested this code in the gis library to allow Python Exception
calls in case of a fatal error:

{{{
static jmp_buf jbuf;

static void error_handler(void *arg)
{
     longjmp(jbuf, 1);
}

int call_with_catch(void (*func)(void *), void *arg)
{
     if (setjmp(jbuf) == 0) {
         G_add_error_handler(error_handler, NULL);
         (*func)(arg);
         G_remove_error_handler(error_handler, NULL);
         return 0;
     }
     else {
         G_remove_error_handler(error_handler, NULL);
         return -1;
     }
}
}}}

Trying to implement this conception, i struggled with converting multiple
function arguments using void pointer handling in ctypes. The code below
works only for specific functions. This file is called ''catch.c'' located
in the lib/gis directory:

{{{
#include <setjmp.h>
#include <grass/gis.h>

static jmp_buf jbuf;

static void error_handler (void *arg)
{
  longjmp (jbuf, 1);
}

int G_call_with_catch (int (*func) (const char*, const char*), const char
*name, const char *mapset, int *state)
{
  if (setjmp (jbuf) == 0)
    {
      int ret;
      G_add_error_handler (error_handler, NULL);
      ret = (*func) (name, mapset);
      G_remove_error_handler (error_handler, NULL);
      *state = 0;
      return ret;

    }
  else
    {
      G_remove_error_handler (error_handler, NULL);
      *state = -1;
      return 0;
    }
}
}}}
The entry in ''include/defs/gis.h'':
{{{
int G_call_with_catch (int (*func) (const char*, const char*), const char
*, const char *, int *);
}}}
The Python code to catch the fatal error call of Rast_open_old:
{{{
import grass.lib.gis as gis
import grass.lib.raster as raster
from ctypes import *

ropen = CFUNCTYPE(c_int, c_char_p, c_char_p)(raster.Rast_open_old)

state = c_int()

fd = gis.G_call_with_catch(ropen, "raster_float", "PERMANENT",
byref(state))

if state.value != 0:
        raise Exception("Error")
}}}

The problem is that the wrapped library functions have all kind/types of
return values and arguments. Trying to catch this in Python using ctypes
is far beyond my capabilities and IMHO tricky.

My suggestion is to generate a wrapper around each function which may call
fatal error, using the setjmp/longjmp approach from Glynn. Example:

The raster open function
{{{
int Rast_open_old(char *name, char* mapset);
}}}
Will be wrapped by this function
{{{
/** This function will call Rast_open_old() and catch the exit call
  * in case a fatal error occurs.
  *
  * \param state This variable is set to 0 on success and -1 in case of a
fatal error
  * \param message This variable must be large enough to store the fatal
error message
  */
int Rast_open_old_noexit(char *name, char *mapset, int *state, char
*message){
/* doing setjmp stuff here, setting and unsetting error handler, ... */
}
}}}
Python code using this wrapper may look like this
{{{
import grass.lib.gis as gis
import grass.lib.raster as raster
from ctypes import *

name = "elevation"
mapset = "PERMANENT"
state = c_int()
message = 2048 * c_char

fd = raster.Rast_open_old_noexit(name, mapset, byref(state),
byref(message))
if state != 0:
     raise Exception("Fatal error message: %s"%message)
}}}

Such wrapping functions can be generated automatically in a pre-compile
process in each library directory. Each function name will be extended
with a ''_noexit'' prefix and two new variables will be added: ''state''
and ''message''. A simple Python script can generate the wrapper and
includes files.

Any suggestions are welcome.

Soeren

[1] http://comments.gmane.org/gmane.comp.gis.grass.devel/47721

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1646&gt;
GRASS GIS <http://grass.osgeo.org>

#1646: GRASS ctypes exception handling
-----------------------------------------------------+----------------------
Reporter: huhabla | Owner: grass-dev@…
     Type: enhancement | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python ctypes | Version: svn-trunk
Keywords: setjmp, longjmp, Exception, gently exit | Platform: All
      Cpu: All |
-----------------------------------------------------+----------------------

Comment(by glynn):

Replying to [ticket:1646 huhabla]:
One way to get around the type issue is to define a call_with_catch
function which is callable from Python, as described under
[http://docs.python.org/extending/extending.htm Extending and Embedding
the Python Interpreter]. The function would have the same signature as
Python's apply() function, but would catch fatal errors and convert them
to Python exceptions.

If you only need coarse-grained fatal-error handling (e.g. calling exit
handlers), then you don't need to wrap every function, just the script's
main() function. Actually, you don't even need that; you can use the
ctypes wrapper for G_add_error_handler(), just as a C module would.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1646#comment:1&gt;
GRASS GIS <http://grass.osgeo.org>

#1646: GRASS ctypes exception handling
-----------------------------------------------------+----------------------
Reporter: huhabla | Owner: grass-dev@…
     Type: enhancement | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python ctypes | Version: svn-trunk
Keywords: setjmp, longjmp, Exception, gently exit | Platform: All
      Cpu: All |
-----------------------------------------------------+----------------------

Comment(by glynn):

Replying to [comment:1 glynn]:

> If you only need coarse-grained fatal-error handling (e.g. calling exit
handlers),

And, in fact, that's all that should be attempted. Fine-grained error
handling is largely pointless given that you cannot safely call GRASS
library functions once a fatal error has been triggered (library functions
are not required to ensure that internal data structures are in a
consistent state before generating a fatal error).

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1646#comment:2&gt;
GRASS GIS <http://grass.osgeo.org>

#1646: GRASS ctypes exception handling
-----------------------------------------------------+----------------------
Reporter: huhabla | Owner: grass-dev@…
     Type: enhancement | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python ctypes | Version: svn-trunk
Keywords: setjmp, longjmp, Exception, gently exit | Platform: All
      Cpu: All |
-----------------------------------------------------+----------------------

Comment(by huhabla):

Replying to [comment:2 glynn]:
> Replying to [comment:1 glynn]:
>
> > If you only need coarse-grained fatal-error handling (e.g. calling
exit handlers),
>
> And, in fact, that's all that should be attempted. Fine-grained error
handling is largely pointless given that you cannot safely call GRASS
library functions once a fatal error has been triggered (library functions
are not required to ensure that internal data structures are in a
consistent state before generating a fatal error).

Ok, i will try to set Python error handler to clean up in the modules and
python libraries. I still struggle with the void pointer handling and
Python object casting. A simple example:

{{{
import grass.lib.gis as gis
import grass.lib.raster as raster
from ctypes import *

class FatalErrorException(Exception):
         pass

def error_routine(message, flag):
     raise FatalErrorException(message)

def cleanup_handler(name):
     print "Removing %s"%(name)

ERROR_ROUTINE = CFUNCTYPE(c_int, c_char_p, c_int)(error_routine)
CLEANUP_HANDLER = CFUNCTYPE(c_void_p, c_void_p)(cleanup_handler)

def set_raise_on_library_error_routine(raise_exp = True):
     if raise_exp:
         gis.G_set_error_routine(ERROR_ROUTINE)
     else:
         gis.G_unset_error_routine()

def main():
     gis.G_gisinit("test")

     set_raise_on_library_error_routine(True)
     name = "elevation_not_exists"
     mapset = "PERMANENT"

     gis.G_add_error_handler(CLEANUP_HANDLER, "%s@%s"%(name, mapset))
     fd = raster.Rast_open_old(name, mapset)

if __name__ == "__main__":
     main()
}}}

Running this module will result in:

{{{
GRASS 7.0.svn (Test):~/src > python test.py
Traceback (most recent call last):
   File "_ctypes/callbacks.c", line 313, in 'calling callback function'
   File "test.py", line 9, in error_routine
     raise FatalErrorException(message)
__main__.FatalErrorException: Raster map <elevation_not_exists@PERMANENT>
not found
Removing 10575748
}}}

Raising a FatalErrorException is a bit pointless, but it works as
expected. The main problem is that i have no clue how to cast the Python
string from void pointer in the clean up handler, or how to cast any other
Python object i need to clean up?

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1646#comment:3&gt;
GRASS GIS <http://grass.osgeo.org>

#1646: GRASS ctypes exception handling
-----------------------------------------------------+----------------------
Reporter: huhabla | Owner: grass-dev@…
     Type: enhancement | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python ctypes | Version: svn-trunk
Keywords: setjmp, longjmp, Exception, gently exit | Platform: All
      Cpu: All |
-----------------------------------------------------+----------------------

Comment(by glynn):

Replying to [comment:3 huhabla]:

> A simple example:
{{{
CLEANUP_HANDLER = CFUNCTYPE(c_void_p, c_void_p)(cleanup_handler)
}}}

I think that this should be

{{{
CLEANUP_HANDLER = CFUNCTYPE(None, c_void_p)(cleanup_handler)
}}}

The return type should be "void", not "void *".

> Raising a !FatalErrorException is a bit pointless, but it works as
expected. The main problem is that i have no clue how to cast the Python
string from void pointer in the clean up handler, or how to cast any other
Python object i need to clean up?

I think that the handler should just call sys.exit(), so that any exit
handlers registered with atexit.register() are called.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1646#comment:4&gt;
GRASS GIS <http://grass.osgeo.org>

#1646: GRASS ctypes exception handling
----------------------------+-----------------------------------------------
  Reporter: huhabla | Owner: grass-dev@…
      Type: enhancement | Status: closed
  Priority: normal | Milestone: 7.0.0
Component: Python ctypes | Version: svn-trunk
Resolution: wontfix | Keywords: setjmp, longjmp, Exception, gently exit
  Platform: All | Cpu: All
----------------------------+-----------------------------------------------
Changes (by wenzeslaus):

  * status: new => closed
  * resolution: => wontfix

Comment:

What is the current status and opinion on this? Does RPC approach (with
Python multiprocessing) from #2134 solves this issue or does it at least
say how this should be implemented? One cleanup handler with
`FatalErrorException` or wrapper for each function exposed by ctypes sound
interesting (and also some C and C++ applications might appreciate some
`set_error_routine` but we already had this discussion). According to
commits related to #2134, I would say that this is wontfix, so I'm closing
this ticket (reopen if I'm mistaken).

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/1646#comment:5&gt;
GRASS GIS <http://grass.osgeo.org>