[GRASS-dev] [GRASS GIS] #2902: i.segment.hierrachical: Execution of subprocesses was not successful

GDAL · February 9, 2016, 2:20pm

#2902: i.segment.hierrachical: Execution of subprocesses was not successful
-------------------------------------------------+-------------------------
Reporter: mlennert | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.4
Component: Addons | Version: svn-trunk
Keywords: i.segment.hierarchical GridModule | CPU: Unspecified
subprocesses |
Platform: Unspecified |
-------------------------------------------------+-------------------------
Using the NC demo dataset, I have the following issue with
i.segment.hierarchical (in fact the GridModule):

{{{
g.region rast=ortho_2001_t792_1m
i.group ortho in=ortho_2001_t792_1m
i.segment.hierarchical group=ortho thresholds=0.01,0.02,0.03
output=seg_ortho_out outputs_pref=seg_ortho memory=2000 width=1524
height=1524 processes=4
}}}

Result:

{{{
Traceback (most recent call last):
   File "/home/mlennert/.grass7/addons/scripts/i.segment.hierarchical",
line 252, in <module>
     SEG.run()
   File "/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/pygrass/modules/grid/grid.py", line 553, in run
     raise RuntimeError(_("Execution of subprocesses was not successful"))
RuntimeError: L'exécution du sous-module a échoué
GRASS 7.1.svn (nc_spm_08):/data/home/mlennert > i.segment.hierarchical
group=ortho thresholds=0.01,0.02,0.03 output=seg_ortho_out
outputs_pref=seg_ortho memory=2000 width=1524 height=1524 processes=4
Traceback (most recent call last):
   File "/home/mlennert/.grass7/addons/scripts/i.segment.hierarchical",
line 252, in <module>
     SEG.run()
   File "/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/pygrass/modules/grid/grid.py", line 553, in run
     raise RuntimeError(_("Execution of subprocesses was not successful"))
RuntimeError: L'exécution du sous-module a échoué
}}}

Anna already reported such an error on the mailing list
[https://lists.osgeo.org/pipermail/grass-dev/2015-November/077357.html 1],
but I haven't seen a response. And contrary to what she reports, I have
this problem systematically, not in random manner.

I also tried reducing the number of processes, but to no avail.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2902>
GRASS GIS <https://grass.osgeo.org>

GDAL · September 6, 2016, 5:31pm

#2902: i.segment.hierarchical: Execution of subprocesses was not successful
-------------------------+-------------------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.5
Component: Addons | Version: svn-trunk
Resolution: | Keywords: i.segment.hierarchical GridModule
       CPU: | subprocesses
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------

Comment (by mlennert):

I still have the same problem

{{{
i.group ls_ms
in=lsat7_2002_10@PERMANENT,lsat7_2002_30@PERMANENT,lsat7_2002_20@PERMANENT,lsat7_2002_40@PERMANENT,lsat7_2002_50@PERMANENT,lsat7_2002_70@PERMANENT
g.region rast=lsat7_2002_70@PERMANENT
i.segment.hierarchical ls_ms threshold=0.1 output=ls_ms_seg width=200
height=200 overlap=10
}}}

results in

{{{
Traceback (most recent call last):
   File "/home/mlennert/.grass7/addons/scripts/i.segment.hierarchical",
line 252, in <module>
     SEG.run()
   File "/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/pygrass/modules/grid/grid.py", line 553, in run
     raise RuntimeError(_("Execution of subprocesses was not successful"))
RuntimeError: Execution of subprocesses was not successful
}}}

I don't really know how to start debugging this. Could someone point me in
the right direction ?

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2902#comment:2>
GRASS GIS <https://grass.osgeo.org>

GDAL · September 6, 2016, 7:40pm

#2902: i.segment.hierarchical: Execution of subprocesses was not successful
-------------------------+-------------------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.5
Component: Addons | Version: svn-trunk
Resolution: | Keywords: i.segment.hierarchical GridModule
       CPU: | subprocesses
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------

Comment (by annakrat):

I remember GridModule had a debug parameter, which switched off the
parallelization, so you start there.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2902#comment:3>
GRASS GIS <https://grass.osgeo.org>

GDAL · September 7, 2016, 9:36am

#2902: i.segment.hierarchical: Execution of subprocesses was not successful
-------------------------+-------------------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.5
Component: Addons | Version: svn-trunk
Resolution: | Keywords: i.segment.hierarchical GridModule
       CPU: | subprocesses
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------

Comment (by mlennert):

Replying to [comment:3 annakrat]:
> I remember GridModule had a debug parameter, which switched off the
parallelization, so you start there.

Yes ! And Pietro actually put a nice big DEBUG=False at the beginning of
the code. Blind me just didn't see it. Setting this to True led me to a
bug in pygrass which still had a call to type='rast' instead of
type='raster'. Corrected in trunk in r69392. But I get new errors so I'll
work on that before I backport to the other grass7 branches.

Thanks for the pointer !

Moritz

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2902#comment:4>
GRASS GIS <https://grass.osgeo.org>

neteler · September 7, 2016, 10:07am

On Wed, Sep 7, 2016 at 11:36 AM, GRASS GIS <trac@osgeo.org> wrote:

#2902: i.segment.hierarchical: Execution of subprocesses was not successful
…
Setting this to True led me to a
bug in pygrass which still had a call to type=‘rast’ instead of
type=‘raster’. Corrected in trunk in r69392.

There appear to be some more to be fixed but I cannot say which ones
are false positives in this sloppy grep search (sorry for the HTML formatting to preserve line breaks):

find . -type f | xargs grep “‘rast’” | grep -v svn/pristine

./scripts/r.shade/r.shade.py: type=‘rast’, name=maps)
./scripts/r.grow/r.grow.py: type=‘rast’, name=map)
./gui/wxpython/psmap/dialogs.py: {‘rast’: self.currRaster, ‘type’: rasterType})
./gui/wxpython/psmap/dialogs.py: {‘rast’: currRaster, ‘type’: str(rasterType)})
./gui/wxpython/lmgr/layertree.py: module = ‘rast’
./lib/python/gunittest/gutils.py: if type == ‘rast’ or type == ‘raster’:
./lib/python/pygrass/raster/abstract.py: utils.remove(self.name, ‘rast’)
./lib/python/pygrass/raster/abstract.py: utils.rename(self.name, newname, ‘rast’)
./lib/python/pygrass/modules/grid/grid.py: for r in findmaps(‘rast’, location=dst[1], gisdbase=dst[2])]
./lib/python/pygrass/modules/interface/testsuite/test_parameter.py: param = Parameter(diz=dict(name=‘rast’, required=‘yes’,
./lib/python/pygrass/modules/interface/testsuite/test_parameter.py: param = Parameter(diz=dict(name=‘rast’, required=‘yes’,
./lib/python/temporal/temporal_algebra.py: maptype=‘rast’, mapclass=RasterDataset,
./lib/python/script/core.py: if element == ‘raster’ or element == ‘rast’:

find . -type f | xargs grep “‘vect’” | grep -v svn/pristine
./scripts/v.db.reconnect.all/v.db.reconnect.all.py: vectors = gscript.list_grouped(‘vect’)[mapset]
./scripts/v.build.all/v.build.all.py: vectors = grass.list_grouped(‘vect’)[mapset]
./gui/wxpython/lmgr/layertree.py: module = ‘vect’
./lib/python/gunittest/gutils.py: elif type == ‘vect’:
./lib/python/pygrass/utils.py: >>> remove(‘test_vect_2’,‘vect’)
./lib/python/pygrass/vector/table.py: >>> copy(test_vector_name,‘mycensus’,‘vect’)
./lib/python/pygrass/vector/table.py: >>> remove(‘mycensus’, ‘vect’)
./lib/python/pygrass/vector/table.py: >>> copy(test_vector_name,‘mycensus’,‘vect’)
./lib/python/pygrass/vector/table.py: >>> remove(‘mycensus’, ‘vect’)
./lib/python/pygrass/vector/table.py: >>> copy(test_vector_name,‘mycensus’,‘vect’)
./lib/python/pygrass/vector/table.py: >>> remove(‘mycensus’, ‘vect’)
./lib/python/pygrass/vector/table.py: >>> copy(test_vector_name,‘mycensus’,‘vect’)
./lib/python/pygrass/vector/table.py: >>> remove(‘mycensus’,‘vect’)
./lib/python/pygrass/vector/abstract.py: utils.rename(self.name, newname, ‘vect’)
./lib/python/pygrass/vector/abstract.py: utils.remove(self.name, ‘vect’)
./lib/python/pygrass/vector/init.py: >>> copy(test_vector_name,‘mytest_vect’,‘vect’)
./lib/python/pygrass/vector/init.py: >>> remove(‘mytest_vect’, ‘vect’)
./lib/python/script/core.py: >>> list_grouped(‘vect’, pattern=‘roads’)[‘PERMANENT’]
./lib/python/script/db.py: vects = list_strings(‘vect’)

… anyone willing to go through and fix the few true Python issues?

Those should then also be backported to 7.2.svn.

thanks
Markus

mlennert · September 7, 2016, 8:00pm

On 07/09/16 12:07, Markus Neteler wrote:

On Wed, Sep 7, 2016 at 11:36 AM, GRASS GIS <trac@osgeo.org
<mailto:trac@osgeo.org>> wrote:

#2902: i.segment.hierarchical: Execution of subprocesses was not

successful
...

Setting this to True led me to a
bug in pygrass which still had a call to type='rast' instead of
type='raster'. Corrected in trunk in r69392.

There appear to be some more to be fixed but I cannot say which ones
are false positives in this sloppy grep search (sorry for the HTML
formatting to preserve line breaks):

find . -type f | xargs grep "'rast'" | grep -v svn/pristine
./scripts/r.shade/r.shade.py <http://r.shade.py>:
type='rast', name=maps)
./scripts/r.grow/r.grow.py
<http://r.grow.py>: type='rast', name=map)
./gui/wxpython/psmap/dialogs.py: {'rast': self.currRaster,
'type': rasterType})
./gui/wxpython/psmap/dialogs.py: {'rast': currRaster, 'type':
str(rasterType)})
./gui/wxpython/lmgr/layertree.py: module = 'rast'
./lib/python/gunittest/gutils.py: if type == 'rast' or type == 'raster':
./lib/python/pygrass/raster/abstract.py: utils.remove(self.name
<http://self.name>, 'rast')
./lib/python/pygrass/raster/abstract.py:
utils.rename(self.name <http://self.name>, newname, 'rast')
./lib/python/pygrass/modules/grid/grid.py: for r in
findmaps('rast', location=dst[1], gisdbase=dst[2])]
./lib/python/pygrass/modules/interface/testsuite/test_parameter.py:
param = Parameter(diz=dict(name='rast', required='yes',
./lib/python/pygrass/modules/interface/testsuite/test_parameter.py:
param = Parameter(diz=dict(name='rast', required='yes',
./lib/python/temporal/temporal_algebra.py: maptype='rast',
mapclass=RasterDataset,
./lib/python/script/core.py: if element == 'raster' or element == 'rast':

find . -type f | xargs grep "'vect'" | grep -v svn/pristine
./scripts/v.db.reconnect.all/v.db.reconnect.all.py
<http://v.db.reconnect.all.py>: vectors =
gscript.list_grouped('vect')[mapset]
./scripts/v.build.all/v.build.all.py <http://v.build.all.py>: vectors
= grass.list_grouped('vect')[mapset]
./gui/wxpython/lmgr/layertree.py: module = 'vect'
./lib/python/gunittest/gutils.py: elif type == 'vect':
./lib/python/pygrass/utils.py: >>> remove('test_vect_2','vect')
./lib/python/pygrass/vector/table.py: >>>
copy(test_vector_name,'mycensus','vect')
./lib/python/pygrass/vector/table.py: >>> remove('mycensus', 'vect')
./lib/python/pygrass/vector/table.py: >>>
copy(test_vector_name,'mycensus','vect')
./lib/python/pygrass/vector/table.py: >>> remove('mycensus', 'vect')
./lib/python/pygrass/vector/table.py: >>>
copy(test_vector_name,'mycensus','vect')
./lib/python/pygrass/vector/table.py: >>> remove('mycensus', 'vect')
./lib/python/pygrass/vector/table.py: >>>
copy(test_vector_name,'mycensus','vect')
./lib/python/pygrass/vector/table.py: >>> remove('mycensus','vect')
./lib/python/pygrass/vector/abstract.py:
utils.rename(self.name <http://self.name>, newname, 'vect')
./lib/python/pygrass/vector/abstract.py: utils.remove(self.name
<http://self.name>, 'vect')
./lib/python/pygrass/vector/__init__.py: >>>
copy(test_vector_name,'mytest_vect','vect')
./lib/python/pygrass/vector/__init__.py: >>>
remove('mytest_vect', 'vect')
./lib/python/script/core.py: >>> list_grouped('vect',
pattern='*roads*')['PERMANENT']
./lib/python/script/db.py: vects = list_strings('vect')

... anyone willing to go through and fix the few true Python issues?

Those should then also be backported to 7.2.svn.

I'd rather have some of the real pygrass and script library experts look at this, but AFAICS, none of these (except for grid.py call to findmaps (but which I already fixed in trunk) actually cause any trouble. I think it was the specfic findmaps function call that uses a ctypes call that caused the issue.

But maybe someone else could confirm or infirm this ?

Moritz

GDAL · September 8, 2016, 8:08am

#2902: i.segment.hierarchical: Execution of subprocesses was not successful
-------------------------+-------------------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.5
Component: Addons | Version: svn-trunk
Resolution: | Keywords: i.segment.hierarchical GridModule
       CPU: | subprocesses
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------

Comment (by mlennert):

The first error was solved, but now I get a new error (same message) at
the very end of an attempt to segment a fairly large image (400 mio
pixels), i.e. after the patching of all tiles. Unfortunately, I do not get
the error when setting the debug parameter, so I cannot see what exactly
is the issue.

Any hints ?

Moritz

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2902#comment:5>
GRASS GIS <https://grass.osgeo.org>

GDAL · September 9, 2016, 7:57am

#2902: i.segment.hierarchical: Execution of subprocesses was not successful
-------------------------+-------------------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.5
Component: Addons | Version: svn-trunk
Resolution: | Keywords: i.segment.hierarchical GridModule
       CPU: | subprocesses
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------

Comment (by mlennert):

Replying to [comment:4 mlennert]:
> Replying to [comment:3 annakrat]:
> > I remember GridModule had a debug parameter, which switched off the
parallelization, so you start there.
>
> Yes ! And Pietro actually put a nice big DEBUG=False at the beginning of
the code. Blind me just didn't see it. Setting this to True led me to a
bug in pygrass which still had a call to type='rast' instead of
type='raster'. Corrected in trunk in r69392.

Any objections against backporting this small change, to the two release
branches ?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2902#comment:6>
GRASS GIS <https://grass.osgeo.org>

GDAL · September 9, 2016, 8:05am

#2902: i.segment.hierarchical: Execution of subprocesses was not successful
-------------------------+-------------------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.5
Component: Addons | Version: svn-trunk
Resolution: | Keywords: i.segment.hierarchical GridModule
       CPU: | subprocesses
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------

Comment (by mlennert):

Replying to [comment:5 mlennert]:
> The first error was solved, but now I get a new error (same message) at
the very end of an attempt to segment a fairly large image (400 mio
pixels), i.e. after the patching of all tiles. Unfortunately, I do not get
the error when setting the debug parameter, so I cannot see what exactly
is the issue.
>

I got a more concrete message:

{{{
Start patching the segments
Traceback (most recent call last):
   File "/home/mlennert/.grass7/addons/scripts/i.segment.hierarchical",
line 254, in <module>
     SEG.run()
   File "/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/pygrass/modules/grid/grid.py", line 567, in run
     self.patch()
   File "/home/mlennert/.grass7/addons/scripts/i.segment.hierarchical",
line 182, in patch
     self.start_row, self.start_col, self.out_prefix)
   File
"/home/mlennert/.grass7/addons/etc/i.segment.hierarchical/isegpatch.py",
line 46, in rpatch_map
     rrasts[-1].open('r')
   File "/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/pygrass/raster/__init__.py", line 210, in open
     raise OpenError(str_err)
grass.exceptions.OpenError: La carte n'existe pas, impossible de l'ouvrir
en mode 'r'
}}}

Sorry for the French. In English it is: "The map does not exist, I can't
open in 'r' mode"

As indicated, this comes from the open() function of the RasterRow class
in lib/python/pygrass/raster/__init__.py, which is called in the
iseg_patch.py file of i.segment.hierarchical:

{{{
         for col in range(len(rbbox)):
             msgr.percent(row, nrows, 1)
             rrasts.append(RasterRow(name=raster,
                                     mapset=mset_str % (start_row + row,
                                                        start_col + col)))
             rrasts[-1].open('r')
}}}

At this stage, I don't really understand which file it is and why it
doesn't exist...

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2902#comment:7>
GRASS GIS <https://grass.osgeo.org>

GDAL · September 9, 2016, 8:16am

#2902: i.segment.hierarchical: Execution of subprocesses was not successful
-------------------------+-------------------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.5
Component: Addons | Version: svn-trunk
Resolution: | Keywords: i.segment.hierarchical GridModule
       CPU: | subprocesses
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------

Comment (by neteler):

Replying to [comment:6 mlennert]:
> Replying to [comment:4 mlennert]:
> > Yes ! And Pietro actually put a nice big DEBUG=False at the beginning
of the code. Blind me just didn't see it. Setting this to True led me to a
bug in pygrass which still had a call to type='rast' instead of
type='raster'. Corrected in trunk in r69392.
>
> Any objections against backporting this small change, to the two release
branches ?

Done in r69410 and r69416.

Generally I think that such bugfixes can be backported right away.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2902#comment:8>
GRASS GIS <https://grass.osgeo.org>

GDAL · September 12, 2016, 9:02am

#2902: i.segment.hierarchical: Execution of subprocesses was not successful
-------------------------+-------------------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.5
Component: Addons | Version: svn-trunk
Resolution: | Keywords: i.segment.hierarchical GridModule
       CPU: | subprocesses
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------

Comment (by mlennert):

Replying to [comment:7 mlennert]:
> Replying to [comment:5 mlennert]:
> > The first error was solved, but now I get a new error (same message)
at the very end of an attempt to segment a fairly large image (400 mio
pixels), i.e. after the patching of all tiles. Unfortunately, I do not get
the error when setting the debug parameter, so I cannot see what exactly
is the issue.
> >
>
> I got a more concrete message:
>
> {{{
> Start patching the segments
> Traceback (most recent call last):
> File "/home/mlennert/.grass7/addons/scripts/i.segment.hierarchical",
line 254, in <module>
> SEG.run()
> File "/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/pygrass/modules/grid/grid.py", line 567, in run
> self.patch()
> File "/home/mlennert/.grass7/addons/scripts/i.segment.hierarchical",
line 182, in patch
> self.start_row, self.start_col, self.out_prefix)
> File
"/home/mlennert/.grass7/addons/etc/i.segment.hierarchical/isegpatch.py",
line 46, in rpatch_map
> rrasts[-1].open('r')
> File "/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-
gnu/etc/python/grass/pygrass/raster/__init__.py", line 210, in open
> raise OpenError(str_err)
> grass.exceptions.OpenError: La carte n'existe pas, impossible de
l'ouvrir en mode 'r'
> }}}
>
> Sorry for the French. In English it is: "The map does not exist, I can't
open in 'r' mode"
>
> As indicated, this comes from the open() function of the RasterRow class
in lib/python/pygrass/raster/__init__.py, which is called in the
iseg_patch.py file of i.segment.hierarchical:
>
> {{{
> for col in range(len(rbbox)):
> msgr.percent(row, nrows, 1)
> rrasts.append(RasterRow(name=raster,
> mapset=mset_str % (start_row + row,
> start_col +
col)))
> rrasts[-1].open('r')
> }}}
>
> At this stage, I don't really understand which file it is and why it
doesn't exist...

Just pure speculation at this point: As this error arrives only when
working on a large region, which means that if tiles are small, there are
many tiles, could it be that rrasts becomes too big ?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2902#comment:9>
GRASS GIS <https://grass.osgeo.org>

GDAL · September 26, 2016, 9:03am

#2902: i.segment.hierarchical: Execution of subprocesses was not successful
-------------------------+-------------------------------------------------
  Reporter: mlennert | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: normal | Milestone: 7.0.5
Component: Addons | Version: svn-trunk
Resolution: fixed | Keywords: i.segment.hierarchical GridModule
       CPU: | subprocesses
  Unspecified | Platform: Unspecified
-------------------------+-------------------------------------------------
Changes (by mlennert):

* status: new => closed
* resolution: => fixed

Comment:

As the original bug is fixed and I have trouble reproducing the second
issue reliably, I'm closing this bug and will open another one if needed.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2902#comment:10>
GRASS GIS <https://grass.osgeo.org>