[GRASS-dev] [GRASS GIS] #2151: g.gui.* modules which use temporal framework leave processes after exiting

#2151: g.gui.* modules which use temporal framework leave processes after exiting
--------------------------------------------+-------------------------------
Reporter: annakrat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python | Version: svn-trunk
Keywords: g.gui.animation, temporal, RCP | Platform: Linux
      Cpu: All |
--------------------------------------------+-------------------------------
During start, g.gui.animation and g.gui.timeline create 2 subprocesses,
specifically calling tgis.init() does. I suppose these are related to the
RCP server recently implemented. The problem is that these processes are
not finished when the gui module is closed. This is caused by the fact
that g.gui.* modules are called in the background by
[http://trac.osgeo.org/grass/browser/grass/trunk/gui/wxpython/core/utils.py#L1053
GuiModuleMain] function. Is there anything what we can do about it? I
guess it's not a big deal when a user runs the module a few times but for
me it's pretty annoying.

This applies to Linux and probably Mac OSX, too.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2151&gt;
GRASS GIS <http://grass.osgeo.org>

#2151: g.gui.* modules which use temporal framework leave processes after exiting
--------------------------------------------+-------------------------------
Reporter: annakrat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python | Version: svn-trunk
Keywords: g.gui.animation, temporal, RCP | Platform: Linux
      Cpu: All |
--------------------------------------------+-------------------------------

Comment(by huhabla):

This is really strange. The subprocesses created by tgis.init() for the
messenger and the C-interface are marked as daemon processes (they will be
terminated if the parent process will exit). In addition, the destructor
of the messenger and C-interface objects explicitly terminate the
subprocesses.

It seems that booth mechanism do not work with the GUI? AFAIC the
destructors are not called when the GUI process exits. I have no idea why.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2151#comment:1&gt;
GRASS GIS <http://grass.osgeo.org>

#2151: g.gui.* modules which use temporal framework leave processes after exiting
--------------------------------------------+-------------------------------
Reporter: annakrat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python | Version: svn-trunk
Keywords: g.gui.animation, temporal, RCP | Platform: Linux
      Cpu: All |
--------------------------------------------+-------------------------------

Comment(by huhabla):

I tried to solve the problem by registering a function at exit that
explicitly terminates the subprocesses. I added this code to
lib/python/temporal/core.py:

{{{
#!python
import atexit

def stop_subprocesses():
     """!Stop the messenger and C-interface subprocesses
     """
     global message_interface
     global c_library_interface
     if message_interface:
         message_interface.stop()
     if c_library_interface:
         c_library_interface.stop()

atexit.register(stop_subprocesses)
}}}

This works for temporal modules, but not for the GUI.

The registered function is not be called at exit when g.gui.animate
terminates!
How does the GUI framework terminates so that this functionality does not
work?

Besides of that, it seems to me that tgis.init() is called several time
when g.gui.animate is opened and a space time raster dataset is chosen for
visualization. It hope that this can be reduced to a single tgis.init()
call?

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2151#comment:2&gt;
GRASS GIS <http://grass.osgeo.org>

#2151: g.gui.* modules which use temporal framework leave processes after exiting
--------------------------------------------+-------------------------------
Reporter: annakrat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python | Version: svn-trunk
Keywords: g.gui.animation, temporal, RCP | Platform: Linux
      Cpu: All |
--------------------------------------------+-------------------------------

Comment(by huhabla):

All the methods that i have tried are not working in case the GUI module
uses {{{os.fork()}}} to start the module in the background.

Why do we need fork() at all? Unix already support job management in the
shell and it does not work on windows?

If we really need it, can this be implemented in a better way?

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2151#comment:3&gt;
GRASS GIS <http://grass.osgeo.org>

#2151: g.gui.* modules which use temporal framework leave processes after exiting
--------------------------------------------+-------------------------------
Reporter: annakrat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python | Version: svn-trunk
Keywords: g.gui.animation, temporal, RCP | Platform: Linux
      Cpu: All |
--------------------------------------------+-------------------------------

Comment(by annakrat):

Replying to [comment:2 huhabla]:
>
> Besides of that, it seems to me that tgis.init() is called several time
when g.gui.animate is opened and a space time raster dataset is chosen for
visualization. It hope that this can be reduced to a single tgis.init()
call?

Sure, done in r58578.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2151#comment:4&gt;
GRASS GIS <http://grass.osgeo.org>

#2151: g.gui.* modules which use temporal framework leave processes after exiting
--------------------------------------------+-------------------------------
Reporter: annakrat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python | Version: svn-trunk
Keywords: g.gui.animation, temporal, RCP | Platform: Linux
      Cpu: All |
--------------------------------------------+-------------------------------

Comment(by wenzeslaus):

Replying to [comment:3 huhabla]:
> All the methods that i have tried are not working in case the GUI module
uses {{{os.fork()}}} to start the module in the background.

> Why do we need fork() at all? Unix already support job management in the
shell and it does not work on windows?
>

The idea was to start the application in the background (r57388, r57370 -
r57378 launch GUI in the background). There was also a discussion on the
mailing list with reasoning that `g.gui.*` should behave in the same way
as `g.gui`, that these (`g.gui*`) are special cases (''not able to find
link now'').

While [source:grass/trunk/gui/wxpython/core/utils.py#L1053 GuiModuleMain]
is using Python `os.fork`, `g.gui` is using
[source:grass/trunk/lib/gis/parser.c#L796 G_spawn] from `grass/spawn.h`.

When the generated GUI for a module is shown (`v.pack` or `g.region
--ui`), no special functionality is provided and it blocks terminal unless
you use `&`.

> If we really need it, can this be implemented in a better way?

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2151#comment:5&gt;
GRASS GIS <http://grass.osgeo.org>

#2151: g.gui.* modules which use temporal framework leave processes after exiting
--------------------------------------------+-------------------------------
Reporter: annakrat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python | Version: svn-trunk
Keywords: g.gui.animation, temporal, RCP | Platform: Linux
      Cpu: All |
--------------------------------------------+-------------------------------

Comment(by annakrat):

I just tried this python
[http://code.activestate.com/recipes/278731-creating-a-daemon-the-python-
way/ recipe] and it worked for me, the processes are terminated. I tried
to find the difference and it seems that just this line is enough (in the
GuiModuleMain function):

{{{
--- core/utils.py (revision 58561)
+++ core/utils.py (working copy)
@@ -1064,7 +1064,8 @@
          child_pid = os.fork()
          if child_pid == 0:
              mainfn()
- os._exit(0)
+ else:
+ os._exit(0)
      else:
          mainfn()

}}}

Can I apply it safely or can it cause some other troubles?

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2151#comment:6&gt;
GRASS GIS <http://grass.osgeo.org>

#2151: g.gui.* modules which use temporal framework leave processes after exiting
--------------------------------------------+-------------------------------
Reporter: annakrat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python | Version: svn-trunk
Keywords: g.gui.animation, temporal, RCP | Platform: Linux
      Cpu: All |
--------------------------------------------+-------------------------------

Comment(by annakrat):

Replying to [comment:6 annakrat]:
> I just tried this python
[http://code.activestate.com/recipes/278731-creating-a-daemon-the-python-
way/ recipe] and it worked for me, the processes are terminated. I tried
to find the difference and it seems that just this line is enough (in the
GuiModuleMain function):
>
> {{{
> --- core/utils.py (revision 58561)
> +++ core/utils.py (working copy)
> @@ -1064,7 +1064,8 @@
> child_pid = os.fork()
> if child_pid == 0:
> mainfn()
> - os._exit(0)
> + else:
> + os._exit(0)
> else:
> mainfn()
>
> }}}
>
> Can I apply it safely or can it cause some other troubles?

Soeren had the same idea (r58580). Unfortunately, I can't say it's solved
completely. I have still problem with g.gui.timeline. The subprocesses
don't terminate when it's called with parameter input or when it's called
without any parameter but then you display the dataset. When it's just
opened and closed the subprocesses terminate. For g.gui.animation it's
seems ok now. So I am confused what's different in g.gui.timeline. Well,
at least some progress...

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2151#comment:7&gt;
GRASS GIS <http://grass.osgeo.org>

#2151: g.gui.* modules which use temporal framework leave processes after exiting
--------------------------------------------+-------------------------------
Reporter: annakrat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python | Version: svn-trunk
Keywords: g.gui.animation, temporal, RCP | Platform: Linux
      Cpu: All |
--------------------------------------------+-------------------------------

Comment(by huhabla):

It seems to me that the g.gui.timeline child process is still alive after
the fork. The init process is the new parent of the forked child??

I have patched g.gui.timeline a make it a bit faster and to assure at
least the termination of the messenger and C-interface subprocesses of the
temporal framework:

{{{
#!diff
Index: timeline/frame.py

--- timeline/frame.py (Revision 58582)
+++ timeline/frame.py (Arbeitskopie)
@@ -78,7 +78,18 @@
          self._layout()
          self.temporalType = None
          self.unit = None
+ # We create a database interface here to speedup the GUI
+ self.dbif = tgis.SQLDatabaseInterfaceConnection()
+ self.dbif.connect()

+ def __del__(self):
+ """!Close the database interface and stop the messenger and
C-interface
+ subprocesses.
+ """
+ if self.dbif.connected is True:
+ self.dbif.close()
+ tgis.stop_subprocesses()
+
      def _layout(self):
          """!Creates the main panel with all the controls on it:
               * mpl canvas
@@ -145,15 +156,16 @@
          self.timeData = {}
          mode = None
          unit = None
+
          for series in timeseries:
              name = series[0] + '@' + series[1]
              etype = series[2]
              sp = tgis.dataset_factory(etype, name)
- if not sp.is_in_db():
+ if not sp.is_in_db(dbif=self.dbif):
                  GError(self, message=_("Dataset <%s> not found in
temporal database") % (name))
                  return

- sp.select()
+ sp.select(dbif=self.dbif)

              self.timeData[name] = {}
              self.timeData[name]['elementType'] = series[2]
@@ -167,8 +179,8 @@
                  return

              # check topology
- maps = sp.get_registered_maps_as_objects()
- self.timeData[name]['validTopology'] =
sp.check_temporal_topology(maps)
+ maps = sp.get_registered_maps_as_objects(dbif=self.dbif)
+ self.timeData[name]['validTopology'] =
sp.check_temporal_topology(maps=maps, dbif=self.dbif)

              self.timeData[name]['temporalMapType'] = sp.get_map_time() #
point/interval
              self.timeData[name]['unit'] = None # only with relative
@@ -194,7 +206,7 @@
                                  'north', 'south', 'west', 'east'])

              rows = sp.get_registered_maps(columns=columns, where=None,
- order='start_time', dbif=None)
+ order='start_time',
dbif=self.dbif)
              if rows is None:
                  rows =
              for row in rows:
@@ -385,7 +397,7 @@
          @return (mapName, mapset, type)
          """
          validated =
- tDict = tgis.tlist_grouped('stds', group_type=True)
+ tDict = tgis.tlist_grouped('stds', group_type=True,
dbif=self.dbif)
          # nested list with '(map, mapset, etype)' items
          allDatasets = [[[(map, mapset, etype) for map in maps]
                       for etype, maps in etypesDict.iteritems()]
}}}

The patch is not applied yet, since it does not solve the issue that the
child process is still alive. Maybe an explicit os.exit() should be added
to g.gui.timeline? How about an exit button?

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2151#comment:8&gt;
GRASS GIS <http://grass.osgeo.org>

#2151: g.gui.* modules which use temporal framework leave processes after exiting
--------------------------------------------+-------------------------------
Reporter: annakrat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python | Version: svn-trunk
Keywords: g.gui.animation, temporal, RCP | Platform: Linux
      Cpu: All |
--------------------------------------------+-------------------------------

Comment(by annakrat):

Replying to [comment:8 huhabla]:

> The patch is not applied yet, since it does not solve the issue that the
child process is still alive. Maybe an explicit os.exit() should be added
to g.gui.timeline? How about an exit button?

os.exit would close the whole wxGUI when the timeline tool is called from
the wxGUI. Exit button would not solve anything, it just does the same as
the standard x close button or Alt+F4. I tried it to be sure and there is
no difference. I think you can commit your changes, it's improvement.
Could the problem come from the matplotlib? Maybe I am using it in an
incorrect way.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2151#comment:9&gt;
GRASS GIS <http://grass.osgeo.org>

#2151: g.gui.* modules which use temporal framework leave processes after exiting
--------------------------------------------+-------------------------------
Reporter: annakrat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python | Version: svn-trunk
Keywords: g.gui.animation, temporal, RCP | Platform: Linux
      Cpu: All |
--------------------------------------------+-------------------------------

Comment(by huhabla):

Replying to [comment:9 annakrat]:
> Replying to [comment:8 huhabla]:
>
> > The patch is not applied yet, since it does not solve the issue that
the child process is still alive. Maybe an explicit os.exit() should be
added to g.gui.timeline? How about an exit button?
>
> os.exit would close the whole wxGUI when the timeline tool is called
from the wxGUI. Exit button would not solve anything, it just does the
same as the standard x close button or Alt+F4. I tried it to be sure and
there is no difference. I think you can commit your changes, it's
improvement. Could the problem come from the matplotlib? Maybe I am using
it in an incorrect way.

I have applied the patch and included an explicit kill call in the
destructor of TimelineFrame, see r58597. The kill signal will only be send
if g.gui.timeline works in stand-alone mode (hopefully, please cross-check
that).

Killing the process is a crude solution, but i have no better idea how to
avoid orphaned daemon g.gui.timeline processes.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2151#comment:10&gt;
GRASS GIS <http://grass.osgeo.org>

#2151: g.gui.* modules which use temporal framework leave processes after exiting
--------------------------------------------+-------------------------------
Reporter: annakrat | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Python | Version: svn-trunk
Keywords: g.gui.animation, temporal, RCP | Platform: Linux
      Cpu: All |
--------------------------------------------+-------------------------------

Comment(by annakrat):

Replying to [comment:10 huhabla]:
> Replying to [comment:9 annakrat]:
> > Replying to [comment:8 huhabla]:
> >
> > > The patch is not applied yet, since it does not solve the issue that
the child process is still alive. Maybe an explicit os.exit() should be
added to g.gui.timeline? How about an exit button?
> >
> > os.exit would close the whole wxGUI when the timeline tool is called
from the wxGUI. Exit button would not solve anything, it just does the
same as the standard x close button or Alt+F4. I tried it to be sure and
there is no difference. I think you can commit your changes, it's
improvement. Could the problem come from the matplotlib? Maybe I am using
it in an incorrect way.
>
> I have applied the patch and included an explicit kill call in the
destructor of TimelineFrame, see r58597. The kill signal will only be send
if g.gui.timeline works in stand-alone mode (hopefully, please cross-check
that).
>
> Killing the process is a crude solution, but i have no better idea how
to avoid orphaned daemon g.gui.timeline processes.

I found the problem and removed it (r58598, r58599). Now it should
terminate all processes but please check. It was caused by a line of code
which actually did nothing for me but I thought it might be working for
some other version of matplotlib. I should have looked at it earlier. What
about the changes you did? Part of them like the os.kill are redundant
now, not sure about tgis.stop_subprocesses()?

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2151#comment:11&gt;
GRASS GIS <http://grass.osgeo.org>

#2151: g.gui.* modules which use temporal framework leave processes after exiting
-----------------------+----------------------------------------------------
  Reporter: annakrat | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: normal | Milestone: 7.0.0
Component: Python | Version: svn-trunk
Resolution: fixed | Keywords: g.gui.animation, temporal, RCP
  Platform: Linux | Cpu: All
-----------------------+----------------------------------------------------
Changes (by huhabla):

  * status: new => closed
  * resolution: => fixed

Comment:

Replying to [comment:11 annakrat]:
> Replying to [comment:10 huhabla]:
> > Replying to [comment:9 annakrat]:
> > > Replying to [comment:8 huhabla]:
> > >
> > > > The patch is not applied yet, since it does not solve the issue
that the child process is still alive. Maybe an explicit os.exit() should
be added to g.gui.timeline? How about an exit button?
> > >
> > > os.exit would close the whole wxGUI when the timeline tool is called
from the wxGUI. Exit button would not solve anything, it just does the
same as the standard x close button or Alt+F4. I tried it to be sure and
there is no difference. I think you can commit your changes, it's
improvement. Could the problem come from the matplotlib? Maybe I am using
it in an incorrect way.
> >
> > I have applied the patch and included an explicit kill call in the
destructor of TimelineFrame, see r58597. The kill signal will only be send
if g.gui.timeline works in stand-alone mode (hopefully, please cross-check
that).
> >
> > Killing the process is a crude solution, but i have no better idea how
to avoid orphaned daemon g.gui.timeline processes.
>
> I found the problem and removed it (r58598, r58599). Now it should
terminate all processes but please check. It was caused by a line of code
which actually did nothing for me but I thought it might be working for
some other version of matplotlib. I should have looked at it earlier. What
about the changes you did? Part of them like the os.kill are redundant
now, not sure about tgis.stop_subprocesses()?

I kept tgis.stop_subprocesses() within the destructor to assure the
termination of the messenger and C-interface subprocesses when the time
line tool was started from the main GUI. The explicit kill and dependent
code was removed in svn (r58602 and r58609). It works for me, no orphaned
processes after closing g.gui.timeline.

I will close this ticket, marking it as fixed.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2151#comment:12&gt;
GRASS GIS <http://grass.osgeo.org>

#2151: g.gui.* modules which use temporal framework leave processes after exiting
-----------------------+----------------------------------------------------
  Reporter: annakrat | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: normal | Milestone: 7.0.0
Component: Python | Version: svn-trunk
Resolution: fixed | Keywords: g.gui.animation, temporal, RCP
  Platform: Linux | Cpu: All
-----------------------+----------------------------------------------------

Comment(by annakrat):

Replying to [comment:12 huhabla]:
> I will close this ticket, marking it as fixed.

Great, thanks for help

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2151#comment:13&gt;
GRASS GIS <http://grass.osgeo.org>