#3166: Parallelization with tiling for grass.script
------------------------------+-------------------------
Reporter: wenzeslaus | Owner: grass-dev@…
Type: enhancement | Status: new
Priority: normal | Milestone: 7.4.0
Component: Python | Version: unspecified
Keywords: script, parallel | CPU: Unspecified
Platform: Unspecified |
------------------------------+-------------------------
At the same time as r69507, I was working on a simpler approach based on
`grass.script`. At this point, it can do tiling and patching of rasters
and partially 3D rasters. Series of commands is executed on each tile.
This and also the patching runs in parallel. The syntax is very similar to
`grass.script` with a convenient function for tile naming (which is
partially a user responsibility).
Here is an tiling example:
{{{
#!python
# this is the control object
tiled_workflow = TiledWorkflow(nprocs=4, width=500, height=500,
overlap=10)
for namer, workflow in tiled_workflow:
slope = namer.name('raster', 'slope')
aspect = namer.name('raster','aspect')
# now do all as usually, workflow is equivalent of `grass.script`
workflow.run_command('r.slope.aspect', elevation='fractal_surface',
slope=slope)
workflow.run_command('r.slope.aspect', elevation='fractal_surface',
aspect=aspect)
workflow.parse_command('g.region', flags='pg')
# nothing was actually done till now
# do the parallel processing and patching
results = tiled_workflow.execute()
# iterate over the results (here from g.region)
for result in results:
for key, value in result.iteritems():
print key, value
}}}
Example using much smaller portion of the API. Creates list of modules
which are then executed in the background. When import of the `parallel`
module fails, `grass.script` is used instead without any changed in the
main part of the code.
{{{
#!python
try:
from grass.script.parallel import ModuleCallList, execute_by_module
call = ModuleCallList()
parallel = True
except ImportError:
call = gs # fall back to grass.script
parallel = False
for i in range(map_min, map_max + 1):
call.mapcalc(expr, num=i)
if parallel:
execute_by_module(call, nprocs=4)
}}}
The current code uses `r.mapcalc` for patching and PyGRASS code computing
the tiles. One of the main issues with the current code is that it does
not finish when there is an error in the executed module.
--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3166>
GRASS GIS <https://grass.osgeo.org>
