[GRASS-dev] [GRASS GIS] #3197: Memory limit issues with r.univar (when used within i.segment.stats)

#3197: Memory limit issues with r.univar (when used within i.segment.stats)
---------------------+-------------------------
Reporter: jpduffy | Owner: grass-dev@…
     Type: defect | Status: new
Priority: normal | Milestone: 7.0.6
Component: Default | Version: 7.0.4
Keywords: | CPU: x86-32
Platform: Linux |
---------------------+-------------------------
Hello,

I am trying to use i.segment.stats as part of an OBIA workflow on high
resolution raster orthomosaics. So far I have used i.segment.uspo,
i.segment and r.clump to create a layer called gp_seg_optimum_clump. Using
r.clump means that the clumps are sequentially numbered, whereas they are
not directly after i.segment.

I try to run the following command:

i.segment.stats --overwrite --verbose map=gp_seg_optimum_clump@gp1 \
rasters=gp_ortho.1@gp1,gp_ortho.2@gp1,gp_ortho.3@gp1,gp_ortho.4@gp1 \
raster_statistics=min,max,mean,stddev,variance,sum \
csvfile=/home/jpd205/Wales_GRASS/GarronPill/gp_seg_stats \
separator=comma

And I get the following error:

Calculating geometry statistics
Calculating statistics for raster gp_ortho.1@gp1
ERROR: G_realloc: unable to allocate 572000 bytes of memory at
        raster/r.univar/r.univar_main.c:324
Traceback (most recent call last):
   File "/home/jpd205/.grass7/addons/scripts/i.segment.stats", line 251, in
<module>
     main()
   File "/home/jpd205/.grass7/addons/scripts/i.segment.stats", line 176, in
main
     quiet=True)
   File "/usr/lib/grass70/etc/python/grass/script/core.py", line 395, in
run_command
     return handle_errors(returncode, returncode, args, kwargs)
   File "/usr/lib/grass70/etc/python/grass/script/core.py", line 313, in
handle_errors
     returncode=returncode)
grass.exceptions.CalledModuleError: Module run None ['r.univar', '--o', '
--q', '-et', 'zones=gp_seg_optimum_clump@gp1',
'output=/home/jpd205/Wales_GRASS/GarronPill/gp1/.tmp/osgeobeefy/6462.0',
'map=gp_ortho.1@gp1'] ended with error
Process ended with non-zero return code 1. See errors in the (error)
output.

Some details about my region settings and data:

g.region -p
projection: 99 (OSGB 1936 / British National Grid)
zone: 0
datum: osgb36
ellipsoid: airy
north: 208007.00931776
south: 207952.59780698
west: 200993.90853302
east: 201097.28911076
nsres: 0.00430914
ewres: 0.00430914
rows: 12627
cols: 23991
cells: 302934357

r.info map=gp_seg_optimum_clump
+----------------------------------------------------------------------------+
  | Map: gp_seg_optimum_clump Date: Thu Oct 27 14:28:30 2016
|
  | Mapset: gp1 Login of Creator: jpd205
|
  | Location: GarronPill
|
  | DataBase: /home/jpd205/Wales_GRASS
|
  | Title: gp_seg_optimum_clump ( gp_seg_optimum_clump )
|
  | Timestamp: none
|
|----------------------------------------------------------------------------|
  |
|
  | Type of Map: raster Number of Categories: 0
|
  | Data Type: CELL
|
  | Rows: 12627
|
  | Columns: 23991
|
  | Total Cells: 302934357
|
  | Projection: OSGB 1936 / British National Grid
|
  | N: 208007.00931776 S: 207952.59780698 Res: 0.00430914
|
  | E: 201097.28911076 W: 200993.90853302 Res: 0.00430914
|
  | Range of data: min = 1 max = 1264957
|
  |
|
  | Data Source:
|
  | gp_seg_optimum@gp1
|
  |
|
  |
|
  | Data Description:
|
  | generated by r.clump
|
  |
|
  | Comments:
|
  | r.clump --overwrite --verbose -d input="gp_seg_optimum@gp1"
output="\ |
  | gp_seg_optimum_clump" title="gp_seg_optimum_clump"
|
  |
|
+----------------------------------------------------------------------------+

r.info map=gp_ortho.1
+----------------------------------------------------------------------------+
  | Map: gp_ortho.1 Date: Mon Oct 17 09:06:32 2016
|
  | Mapset: gp1 Login of Creator: jd
|
  | Location: GarronPill
|
  | DataBase: /home/jpd205/Wales_GRASS
|
  | Title: ( gp_ortho.1 )
|
  | Timestamp: none
|
|----------------------------------------------------------------------------|
  |
|
  | Type of Map: raster Number of Categories: 0
|
  | Data Type: CELL
|
  | Rows: 12627
|
  | Columns: 23991
|
  | Total Cells: 302934357
|
  | Projection: OSGB 1936 / British National Grid
|
  | N: 208007.00931776 S: 207952.59780698 Res: 0.00430914
|
  | E: 201097.28911076 W: 200993.90853302 Res: 0.00430914
|
  | Range of data: min = 0 max = 255
|
  |
|
  | Data Description:
|
  | generated by r.in.gdal
|
  |
|
  | Comments:
|
  | r.in.gdal -o input="/home/jd/GarronPill_R_BNG_PS2_O_C_RGB_TEX.tif"
o\ |
  | utput="gp_ortho" memory=300 offset=0
|
  |
|
+----------------------------------------------------------------------------+

Moritz Lennert believes that this is a memory issue limit with r.univar.
He suggested I submit this bug report.

Kind regards

James

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3197&gt;
GRASS GIS <https://grass.osgeo.org>

#3197: Memory limit issues with r.univar (when used within i.segment.stats)
----------------------+-------------------------
  Reporter: jpduffy | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.6
Component: Default | Version: 7.0.4
Resolution: | Keywords:
       CPU: x86-32 | Platform: Linux
----------------------+-------------------------
Description changed by martinl:

Old description:

Hello,

I am trying to use i.segment.stats as part of an OBIA workflow on high
resolution raster orthomosaics. So far I have used i.segment.uspo,
i.segment and r.clump to create a layer called gp_seg_optimum_clump.
Using r.clump means that the clumps are sequentially numbered, whereas
they are not directly after i.segment.

I try to run the following command:

i.segment.stats --overwrite --verbose map=gp_seg_optimum_clump@gp1 \
rasters=gp_ortho.1@gp1,gp_ortho.2@gp1,gp_ortho.3@gp1,gp_ortho.4@gp1 \
raster_statistics=min,max,mean,stddev,variance,sum \
csvfile=/home/jpd205/Wales_GRASS/GarronPill/gp_seg_stats \
separator=comma

And I get the following error:

Calculating geometry statistics
Calculating statistics for raster gp_ortho.1@gp1
ERROR: G_realloc: unable to allocate 572000 bytes of memory at
       raster/r.univar/r.univar_main.c:324
Traceback (most recent call last):
  File "/home/jpd205/.grass7/addons/scripts/i.segment.stats", line 251,
in <module>
    main()
  File "/home/jpd205/.grass7/addons/scripts/i.segment.stats", line 176,
in main
    quiet=True)
  File "/usr/lib/grass70/etc/python/grass/script/core.py", line 395, in
run_command
    return handle_errors(returncode, returncode, args, kwargs)
  File "/usr/lib/grass70/etc/python/grass/script/core.py", line 313, in
handle_errors
    returncode=returncode)
grass.exceptions.CalledModuleError: Module run None ['r.univar', '--o', '
--q', '-et', 'zones=gp_seg_optimum_clump@gp1',
'output=/home/jpd205/Wales_GRASS/GarronPill/gp1/.tmp/osgeobeefy/6462.0',
'map=gp_ortho.1@gp1'] ended with error
Process ended with non-zero return code 1. See errors in the (error)
output.

Some details about my region settings and data:

g.region -p
projection: 99 (OSGB 1936 / British National Grid)
zone: 0
datum: osgb36
ellipsoid: airy
north: 208007.00931776
south: 207952.59780698
west: 200993.90853302
east: 201097.28911076
nsres: 0.00430914
ewres: 0.00430914
rows: 12627
cols: 23991
cells: 302934357

r.info map=gp_seg_optimum_clump
+----------------------------------------------------------------------------+
| Map: gp_seg_optimum_clump Date: Thu Oct 27 14:28:30
2016 |
| Mapset: gp1 Login of Creator: jpd205
|
| Location: GarronPill
|
| DataBase: /home/jpd205/Wales_GRASS
|
| Title: gp_seg_optimum_clump ( gp_seg_optimum_clump )
|
| Timestamp: none
|
|----------------------------------------------------------------------------|
|
|
| Type of Map: raster Number of Categories: 0
|
| Data Type: CELL
|
| Rows: 12627
|
| Columns: 23991
|
| Total Cells: 302934357
|
| Projection: OSGB 1936 / British National Grid
|
| N: 208007.00931776 S: 207952.59780698 Res: 0.00430914
|
| E: 201097.28911076 W: 200993.90853302 Res: 0.00430914
|
| Range of data: min = 1 max = 1264957
|
|
|
| Data Source:
|
| gp_seg_optimum@gp1
|
|
|
|
|
| Data Description:
|
| generated by r.clump
|
|
|
| Comments:
|
| r.clump --overwrite --verbose -d input="gp_seg_optimum@gp1"
output="\ |
| gp_seg_optimum_clump" title="gp_seg_optimum_clump"
|
|
|
+----------------------------------------------------------------------------+

r.info map=gp_ortho.1
+----------------------------------------------------------------------------+
| Map: gp_ortho.1 Date: Mon Oct 17 09:06:32
2016 |
| Mapset: gp1 Login of Creator: jd
|
| Location: GarronPill
|
| DataBase: /home/jpd205/Wales_GRASS
|
| Title: ( gp_ortho.1 )
|
| Timestamp: none
|
|----------------------------------------------------------------------------|
|
|
| Type of Map: raster Number of Categories: 0
|
| Data Type: CELL
|
| Rows: 12627
|
| Columns: 23991
|
| Total Cells: 302934357
|
| Projection: OSGB 1936 / British National Grid
|
| N: 208007.00931776 S: 207952.59780698 Res: 0.00430914
|
| E: 201097.28911076 W: 200993.90853302 Res: 0.00430914
|
| Range of data: min = 0 max = 255
|
|
|
| Data Description:
|
| generated by r.in.gdal
|
|
|
| Comments:
|
| r.in.gdal -o input="/home/jd/GarronPill_R_BNG_PS2_O_C_RGB_TEX.tif"
o\ |
| utput="gp_ortho" memory=300 offset=0
|
|
|
+----------------------------------------------------------------------------+

Moritz Lennert believes that this is a memory issue limit with r.univar.
He suggested I submit this bug report.

Kind regards

James

New description:

Hello,

I am trying to use i.segment.stats as part of an OBIA workflow on high
resolution raster orthomosaics. So far I have used i.segment.uspo,
i.segment and r.clump to create a layer called gp_seg_optimum_clump. Using
r.clump means that the clumps are sequentially numbered, whereas they are
not directly after i.segment.

I try to run the following command:

{{{
i.segment.stats --overwrite --verbose map=gp_seg_optimum_clump@gp1 \
rasters=gp_ortho.1@gp1,gp_ortho.2@gp1,gp_ortho.3@gp1,gp_ortho.4@gp1 \
raster_statistics=min,max,mean,stddev,variance,sum \
csvfile=/home/jpd205/Wales_GRASS/GarronPill/gp_seg_stats \
separator=comma
}}}

And I get the following error:

{{{
Calculating geometry statistics
Calculating statistics for raster gp_ortho.1@gp1
ERROR: G_realloc: unable to allocate 572000 bytes of memory at
        raster/r.univar/r.univar_main.c:324
Traceback (most recent call last):
   File "/home/jpd205/.grass7/addons/scripts/i.segment.stats", line 251, in
<module>
     main()
   File "/home/jpd205/.grass7/addons/scripts/i.segment.stats", line 176, in
main
     quiet=True)
   File "/usr/lib/grass70/etc/python/grass/script/core.py", line 395, in
run_command
     return handle_errors(returncode, returncode, args, kwargs)
   File "/usr/lib/grass70/etc/python/grass/script/core.py", line 313, in
handle_errors
     returncode=returncode)
grass.exceptions.CalledModuleError: Module run None ['r.univar', '--o', '
--q', '-et', 'zones=gp_seg_optimum_clump@gp1',
'output=/home/jpd205/Wales_GRASS/GarronPill/gp1/.tmp/osgeobeefy/6462.0',
'map=gp_ortho.1@gp1'] ended with error
Process ended with non-zero return code 1. See errors in the (error)
output.
}}}

Some details about my region settings and data:

{{{
g.region -p
projection: 99 (OSGB 1936 / British National Grid)
zone: 0
datum: osgb36
ellipsoid: airy
north: 208007.00931776
south: 207952.59780698
west: 200993.90853302
east: 201097.28911076
nsres: 0.00430914
ewres: 0.00430914
rows: 12627
cols: 23991
cells: 302934357
}}}

{{{
r.info map=gp_seg_optimum_clump
+----------------------------------------------------------------------------+
  | Map: gp_seg_optimum_clump Date: Thu Oct 27 14:28:30 2016
|
  | Mapset: gp1 Login of Creator: jpd205
|
  | Location: GarronPill
|
  | DataBase: /home/jpd205/Wales_GRASS
|
  | Title: gp_seg_optimum_clump ( gp_seg_optimum_clump )
|
  | Timestamp: none
|
|----------------------------------------------------------------------------|
  |
|
  | Type of Map: raster Number of Categories: 0
|
  | Data Type: CELL
|
  | Rows: 12627
|
  | Columns: 23991
|
  | Total Cells: 302934357
|
  | Projection: OSGB 1936 / British National Grid
|
  | N: 208007.00931776 S: 207952.59780698 Res: 0.00430914
|
  | E: 201097.28911076 W: 200993.90853302 Res: 0.00430914
|
  | Range of data: min = 1 max = 1264957
|
  |
|
  | Data Source:
|
  | gp_seg_optimum@gp1
|
  |
|
  |
|
  | Data Description:
|
  | generated by r.clump
|
  |
|
  | Comments:
|
  | r.clump --overwrite --verbose -d input="gp_seg_optimum@gp1"
output="\ |
  | gp_seg_optimum_clump" title="gp_seg_optimum_clump"
|
  |
|
+----------------------------------------------------------------------------+
}}}

{{{
r.info map=gp_ortho.1
+----------------------------------------------------------------------------+
  | Map: gp_ortho.1 Date: Mon Oct 17 09:06:32 2016
|
  | Mapset: gp1 Login of Creator: jd
|
  | Location: GarronPill
|
  | DataBase: /home/jpd205/Wales_GRASS
|
  | Title: ( gp_ortho.1 )
|
  | Timestamp: none
|
|----------------------------------------------------------------------------|
  |
|
  | Type of Map: raster Number of Categories: 0
|
  | Data Type: CELL
|
  | Rows: 12627
|
  | Columns: 23991
|
  | Total Cells: 302934357
|
  | Projection: OSGB 1936 / British National Grid
|
  | N: 208007.00931776 S: 207952.59780698 Res: 0.00430914
|
  | E: 201097.28911076 W: 200993.90853302 Res: 0.00430914
|
  | Range of data: min = 0 max = 255
|
  |
|
  | Data Description:
|
  | generated by r.in.gdal
|
  |
|
  | Comments:
|
  | r.in.gdal -o input="/home/jd/GarronPill_R_BNG_PS2_O_C_RGB_TEX.tif"
o\ |
  | utput="gp_ortho" memory=300 offset=0
|
  |
|
+----------------------------------------------------------------------------+
}}}

Moritz Lennert believes that this is a memory issue limit with r.univar.
He suggested I submit this bug report.

Kind regards

James

--

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3197#comment:1&gt;
GRASS GIS <https://grass.osgeo.org>

#3197: Memory limit issues with r.univar (when used within i.segment.stats)
----------------------+-------------------------
  Reporter: jpduffy | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.6
Component: Default | Version: 7.0.4
Resolution: | Keywords:
       CPU: x86-32 | Platform: Linux
----------------------+-------------------------

Comment (by zarch):

Replying to [ticket:3197 jpduffy]:
> I get the following error:
>
> {{{
> Calculating geometry statistics
> Calculating statistics for raster gp_ortho.1@gp1
> ERROR: G_realloc: unable to allocate 572000 bytes of memory at
> [...]
> }}}

I had the same problem when I was working OBIA some years ago and I wrote
r.univar2 that it is available as grass-addons.
r.univa2 extract some general statistics from raster using the zones
(consume much less memory than r.univar, and compute more general
statistics like:skewness, kurtosis, and mode. It is available here:
https://trac.osgeo.org/grass/browser/grass-addons/grass7/raster/r.univar2

It was not backported to trunk because it was used only by me and not
tested enough (imho), and If I remember correctly is working only with
zones.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3197#comment:2&gt;
GRASS GIS <https://grass.osgeo.org>

#3197: Memory limit issues with r.univar (when used within i.segment.stats)
----------------------+-------------------------
  Reporter: jpduffy | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.7
Component: Default | Version: 7.0.4
Resolution: | Keywords:
       CPU: x86-32 | Platform: Linux
----------------------+-------------------------

Comment (by martinl):

Still relevant?

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3197#comment:4&gt;
GRASS GIS <https://grass.osgeo.org>

#3197: Memory limit issues with r.univar (when used within i.segment.stats)
----------------------+-------------------------
  Reporter: jpduffy | Owner: grass-dev@…
      Type: defect | Status: new
  Priority: normal | Milestone: 7.0.7
Component: Default | Version: 7.0.4
Resolution: | Keywords:
       CPU: x86-32 | Platform: Linux
----------------------+-------------------------

Comment (by mmetz):

Replying to [comment:4 martinl]:
> Still relevant?

Maybe there is a way to reduce memory consumption per zone, but still,
memory consumption increases with the number of zones. This will probably
not fix.

The addon r.univar2 needs to be reviewed and if regarded as an
improvement, modified to work also without zones.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3197#comment:5&gt;
GRASS GIS <https://grass.osgeo.org>

#3197: Memory limit issues with r.univar (when used within i.segment.stats)
----------------------+-------------------------
  Reporter: jpduffy | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: normal | Milestone: 7.0.7
Component: Default | Version: 7.0.4
Resolution: wontfix | Keywords:
       CPU: x86-32 | Platform: Linux
----------------------+-------------------------
Changes (by martinl):

* status: new => closed
* resolution: => wontfix

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3197#comment:6&gt;
GRASS GIS <https://grass.osgeo.org>