Hi to all,
A new addon is available [1]: i.cutlines tiles the images into tiles with irregular borders that avoid cutting through meaningful objects. This allows tiling an image for parallel processing while avoiding border effects.
Enjoy !
Moritz
[1] https://grass.osgeo.org/grass74/manuals/addons/i.cutlines.html
On Tue, Mar 13, 2018 at 4:06 PM, Moritz Lennert
<mlennert@club.worldonline.be> wrote:
Hi to all,
A new addon is available [1]: i.cutlines tiles the images into tiles with
irregular borders that avoid cutting through meaningful objects. This allows
tiling an image for parallel processing while avoiding border effects.Enjoy !
Cool!
It would also be interesting for post-orthophoto mosaiking.
Markus
Hi Moritz, all,
On Tue, Mar 13, 2018 at 4:06 PM, Moritz Lennert
<mlennert@club.worldonline.be> wrote:
Hi to all,
A new addon is available [1]: i.cutlines tiles the images into tiles with
irregular borders that avoid cutting through meaningful objects. This allows
tiling an image for parallel processing while avoiding border effects.
...
[1] https://grass.osgeo.org/grass74/manuals/addons/i.cutlines.html
we are currently trying to use this approach on a huge (?) area of 50
gigapixels of aerial images (i.e., the free openNRW 10cm orthophotos)
which we have to classify.
In future the areas we'll have to process may be even bigger. Right
now, as far as I understand, i.cutlines is reading the entire images.
This becomes increasingly difficult with bigger areas due to hardware
limitations.
Do you have a suggestion how to deal with that? Kind of chicken and
egg problem? 
thanks
Markus
Hi Markus,
On 13/08/18 13:41, Markus Neteler wrote:
Hi Moritz, all,
On Tue, Mar 13, 2018 at 4:06 PM, Moritz Lennert
<mlennert@club.worldonline.be> wrote:Hi to all,
A new addon is available [1]: i.cutlines tiles the images into tiles with
irregular borders that avoid cutting through meaningful objects. This allows
tiling an image for parallel processing while avoiding border effects....
[1] https://grass.osgeo.org/grass74/manuals/addons/i.cutlines.html
we are currently trying to use this approach on a huge (?) area of 50
gigapixels of aerial images (i.e., the free openNRW 10cm orthophotos)
which we have to classify.In future the areas we'll have to process may be even bigger. Right
now, as far as I understand, i.cutlines is reading the entire images.
This becomes increasingly difficult with bigger areas due to hardware
limitations.Do you have a suggestion how to deal with that? Kind of chicken and
egg problem?
AFAIK, the only moment where i.cutlines potentially reads the whole image would be in the edge detection part. That's why there is the tiling option to avoid just that. So, unless I'm forgetting something, you should be able to work on large images. If you have seen this crash the module, please file a bug report.
Note that for versions < 7.5 (I'll have to check whether Pietro's fix was backported to 7.4) there is a parameter name conflict between i.zc and the GridModule class used for the tiling, so you'd have to use i.edge.
Moritz
Hi Moritz,
On Mon, Aug 13, 2018 at 2:04 PM, Moritz Lennert
<mlennert@club.worldonline.be> wrote:
On 13/08/18 13:41, Markus Neteler wrote:
...
AFAIK, the only moment where i.cutlines potentially reads the whole image
would be in the edge detection part. That's why there is the tiling option
to avoid just that.
I suppose you refer to
tile_width=integer
Width of tiles for tiled edge detection (pixels)
tile_height=integer
Height of tiles for tiled edge detection (pixels)
?
So, unless I'm forgetting something, you should be able
to work on large images. If you have seen this crash the module, please file
a bug report.
OK, fine, we'll try and report.
Note that for versions < 7.5 (I'll have to check whether Pietro's fix was
backported to 7.4)
I'm not sure either.
there is a parameter name conflict between i.zc and the
GridModule class used for the tiling, so you'd have to use i.edge.
At time we use trunk.
thanks,
Markus
On 13/08/18 15:30, Markus Neteler wrote:
Hi Moritz,
On Mon, Aug 13, 2018 at 2:04 PM, Moritz Lennert
<mlennert@club.worldonline.be> wrote:On 13/08/18 13:41, Markus Neteler wrote:
...
AFAIK, the only moment where i.cutlines potentially reads the whole image
would be in the edge detection part. That's why there is the tiling option
to avoid just that.I suppose you refer to
tile_width=integer
Width of tiles for tiled edge detection (pixels)
tile_height=integer
Height of tiles for tiled edge detection (pixels)
?So, unless I'm forgetting something, you should be able
to work on large images. If you have seen this crash the module, please file
a bug report.OK, fine, we'll try and report.
Note that for versions < 7.5 (I'll have to check whether Pietro's fix was
backported to 7.4)I'm not sure either.
Actually you backported it: https://trac.osgeo.org/grass/changeset/72541/grass/branches/releasebranch_7_4/lib/python/pygrass
So, yes, it should work with 7.4.1.
Moritz
On 14/08/18 10:13, Moritz Lennert wrote:
On 13/08/18 15:30, Markus Neteler wrote:
Hi Moritz,
On Mon, Aug 13, 2018 at 2:04 PM, Moritz Lennert
<mlennert@club.worldonline.be> wrote:On 13/08/18 13:41, Markus Neteler wrote:
...
AFAIK, the only moment where i.cutlines potentially reads the whole image
would be in the edge detection part. That's why there is the tiling option
to avoid just that.I suppose you refer to
tile_width=integer
Width of tiles for tiled edge detection (pixels)
tile_height=integer
Height of tiles for tiled edge detection (pixels)
?So, unless I'm forgetting something, you should be able
to work on large images. If you have seen this crash the module, please file
a bug report.
A colleague working with a large dataset > 10e9 pixels) just reported that the module does not crash, but that it seems to take "forever" (he stopped the process after a day). I guess this is in the r.cost phase.
An option would be to tile (and parallelize) the entire process which would mean finding cutlines in the individual tiles, making sure that the start and endpoints of these cutlines match the start and endpoints on the neighboring tiles...
Any suggestions are welcome.
Moritz
On 16/08/18 12:26, Moritz Lennert wrote:
On 14/08/18 10:13, Moritz Lennert wrote:
On 13/08/18 15:30, Markus Neteler wrote:
Hi Moritz,
On Mon, Aug 13, 2018 at 2:04 PM, Moritz Lennert
<mlennert@club.worldonline.be> wrote:On 13/08/18 13:41, Markus Neteler wrote:
...
AFAIK, the only moment where i.cutlines potentially reads the whole image
would be in the edge detection part. That's why there is the tiling option
to avoid just that.I suppose you refer to
tile_width=integer
Width of tiles for tiled edge detection (pixels)
tile_height=integer
Height of tiles for tiled edge detection (pixels)
?So, unless I'm forgetting something, you should be able
to work on large images. If you have seen this crash the module, please file
a bug report.A colleague working with a large dataset > 10e9 pixels) just reported
that the module does not crash, but that it seems to take "forever" (he
stopped the process after a day). I guess this is in the r.cost phase.An option would be to tile (and parallelize) the entire process which
would mean finding cutlines in the individual tiles, making sure that
the start and endpoints of these cutlines match the start and endpoints
on the neighboring tiles...
I see this in r.cost's main.c:
420 /* this is most probably the limitation of r.cost for large datasets
421 * segment size needs to be reduced to avoid unecessary disk IO
422 * but it doesn't make sense to go down to 1
423 * so use 64 segment rows and cols for <= 200 million cells
424 * for larger regions, 32 segment rows and cols
425 * maybe go down to 16 for > 500 million cells ? */
So maybe it's time to experiment a bit more with r.cost in very large regions ?
Moritz