[GRASS-dev] pygrass: How to allow Ctrl-C to stop all subprocesses of GridModule

GRASS GRASS developer
1

And another question on GridModule:

For the same module using GridModule to split the task of edge detection in a raster map into many parallel tasks, when I want to stop the module run using Ctrl-C I get tons of

Process PoolWorker-315:
Traceback (most recent call last):
   File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
   File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
     self.run()
   File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
     self._target(*self._args, **self._kwargs)
   File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
     task = get()
   File "/usr/lib/python2.7/multiprocessing/queues.py", line 374, in get
     task = get()
   File "/usr/lib/python2.7/multiprocessing/queues.py", line 374, in get
     racquire()
KeyboardInterrupt

and cannot stop the module, even when flooding the input with Ctrl-Cs.

Any hint how to allow Ctrl-C to stop all subprocesses ?

Moritz

2

* Moritz Lennert <mlennert@club.worldonline.be> [2018-03-08 15:05:46 +0100]:

And another question on GridModule:

For the same module using GridModule to split the task of edge detection in a raster map into many parallel tasks, when I want to stop the module run using Ctrl-C I get tons of

Process PoolWorker-315:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
   self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
   self._target(*self._args, **self._kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
   task = get()
File "/usr/lib/python2.7/multiprocessing/queues.py", line 374, in get
   task = get()
File "/usr/lib/python2.7/multiprocessing/queues.py", line 374, in get
   racquire()
KeyboardInterrupt

and cannot stop the module, even when flooding the input with Ctrl-Cs.

Any hint how to allow Ctrl-C to stop all subprocesses ?

Moritz

Moritz, could it be a case related to the ones mentioned in

https://stackoverflow.com/a/1635084/1172302
https://stackoverflow.com/a/11816038/1172302

?

Nikos

3

Hi Nikos !

Am 9. März 2018 09:01:25 MEZ schrieb Nikos Alexandris <nik@nikosalexandris.net>:

* Moritz Lennert <mlennert@club.worldonline.be> [2018-03-08 15:05:46
+0100]:

And another question on GridModule:

For the same module using GridModule to split the task of edge
detection in a raster map into many parallel tasks, when I want to
stop the module run using Ctrl-C I get tons of

Process PoolWorker-315:
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in
_bootstrap
File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in
_bootstrap
   self.run()
   self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in

run

File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in

run

   self._target(*self._args, **self._kwargs)
   self._target(*self._args, **self._kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in

worker

File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in

worker

   task = get()
File "/usr/lib/python2.7/multiprocessing/queues.py", line 374, in

get

   task = get()
File "/usr/lib/python2.7/multiprocessing/queues.py", line 374, in

get

   racquire()
   racquire()
KeyboardInterrupt

and cannot stop the module, even when flooding the input with Ctrl-Cs.

Any hint how to allow Ctrl-C to stop all subprocesses ?

Moritz

Moritz, could it be a case related to the ones mentioned in

https://stackoverflow.com/a/1635084/1172302
https://stackoverflow.com/a/11816038/1172302

Related yes, but AFAIU GridModule uses multiprocessing, not multithreading. I think it should be possible to modify something in the GridModule class so that it interrupts, but not sure about what exactly (apparently this is specifically a problem in Python 2. In Python 3 it should happen automatically IIUC.

Moritz

4

* Moritz Lennert <mlennert@club.worldonline.be> [2018-03-09 09:16:01 +0100]:

Hi Nikos !

Am 9. März 2018 09:01:25 MEZ schrieb Nikos Alexandris <nik@nikosalexandris.net>:

* Moritz Lennert <mlennert@club.worldonline.be> [2018-03-08 15:05:46
+0100]:

And another question on GridModule:

For the same module using GridModule to split the task of edge
detection in a raster map into many parallel tasks, when I want to
stop the module run using Ctrl-C I get tons of

Process PoolWorker-315:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in
_bootstrap
File "/usr/lib/python2.7/multiprocessing/process.py", line 267, in
_bootstrap
   self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in

run

File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in

run

   self._target(*self._args, **self._kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in

worker

File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in

worker

   task = get()
File "/usr/lib/python2.7/multiprocessing/queues.py", line 374, in

get

   task = get()
File "/usr/lib/python2.7/multiprocessing/queues.py", line 374, in

get

   racquire()
KeyboardInterrupt

and cannot stop the module, even when flooding the input with Ctrl-Cs.

Any hint how to allow Ctrl-C to stop all subprocesses ?

Moritz

Moritz, could it be a case related to the ones mentioned in

https://stackoverflow.com/a/1635084/1172302
https://stackoverflow.com/a/11816038/1172302

Related yes, but AFAIU GridModule uses multiprocessing, not multithreading. I think it should be possible to modify something in the GridModule class so that it interrupts, but not sure about what exactly (apparently this is specifically a problem in Python 2. In Python 3 it should happen automatically IIUC.

Right. This https://stackoverflow.com/a/35134329/1172302 should be,
however, directly related. However, not portable, as per a comment.

How about this one: https://stackoverflow.com/a/44869451/1172302 (July
2017)? See
also
http://jessenoller.com/blog/2009/01/08/multiprocessingpool-and-keyboardinterrupt.

Nikos

5

Moritz Lennert wrote:

Any hint how to allow Ctrl-C to stop all subprocesses ?

If you can get their PIDs, you should be able to use os.setpgid() to
move them into the same process group as the main script. Ctrl-C (by
default) sends SIGINT to all processes in the foreground process
group.

--
Glynn Clements <glynn@gclements.plus.com>