Thanks all for your thoughts on this. I suspect the problem may be race conditions being set up between https://github.com/qgis/QGIS/blob/master/python/plugins/processing/algs/grass7/Grass7Utils.py and https://github.com/qgis/QGIS/blob/master/python/plugins/processing/algs/grass7/Grass7Algorithm.py. It attempts to use an existing session and shared mapset which could be problematic from the sounds of things. It might be an idea to add an optional parameter to grass algorithms to not use shared mapsets and create a new mapset for each process?
The Grass7 log output from my script is below. The pairs of input rasters for the parallel algorithms should be:
hm1.tif, LS90.tif
hm2.tif, LS14.tif
hm3.tif, HS90.tif
hm4.tif, LS90.tif
If could be that not all of the grass commands were logged, but it looks like HS90.tif and LS14.tif were never passed as parameters to grass. I also added a typical log when the algorithms are executed sequentially (3800-25.tif is reused instead of hm1-4).
Kind regards,
Rudi
2018-08-15T10:09:00 INFO processInputs end. Commands: ['g.proj -c proj4="+proj=aea +lat_1=-24 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs"', 'r.external input="C:\\Users\\rudi\\AppData\\Local\\Temp\\processing_74832af828bb48a9bb0e64fd59d74f24\\cb4c839fbd6f4419873dbc29ffa2b973\\hm4.tif" band=1 output="rast_5b73df9c811b22" --overwrite -o', 'r.external input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\HS14.tif" band=1 output="rast_5b73df9c811b23" --overwrite -o', 'g.region n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308 w=-834195.3236270341 res=92.0757893076434']
2018-08-15T10:09:00 INFO processCommands end. Commands: ['g.proj -c proj4="+proj=aea +lat_1=-24 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs"', 'r.external input="C:\\Users\\rudi\\AppData\\Local\\Temp\\processing_74832af828bb48a9bb0e64fd59d74f24\\cb4c839fbd6f4419873dbc29ffa2b973\\hm4.tif" band=1 output="rast_5b73df9c811b22" --overwrite -o', 'r.external input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\HS14.tif" band=1 output="rast_5b73df9c811b23" --overwrite -o', 'g.region n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308 w=-834195.3236270341 res=92.0757893076434', 'r.stats.zonal base=rast_5b73df9c811b22 cover=rast_5b73df9c811b23 method="average" output=output87eadbc7948b4b5990773af2261f0db8 --overwrite']
2018-08-15T10:09:00 INFO processInputs end. Commands: ['r.external input="C:\\Users\\rudi\\AppData\\Local\\Temp\\processing_74832af828bb48a9bb0e64fd59d74f24\\79ea35021d26468c8220582c6397dd18\\hm1.tif" band=1 output="rast_5b73df9c865914" --overwrite -o', 'r.external input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\LS90.tif" band=1 output="rast_5b73df9c865915" --overwrite -o', 'g.region n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308 w=-834195.3236270341 res=92.0757893076434']
2018-08-15T10:09:00 INFO processCommands end. Commands: ['r.external input="C:\\Users\\rudi\\AppData\\Local\\Temp\\processing_74832af828bb48a9bb0e64fd59d74f24\\79ea35021d26468c8220582c6397dd18\\hm1.tif" band=1 output="rast_5b73df9c865914" --overwrite -o', 'r.external input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\LS90.tif" band=1 output="rast_5b73df9c865915" --overwrite -o', 'g.region n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308 w=-834195.3236270341 res=92.0757893076434', 'r.stats.zonal base=rast_5b73df9c865914 cover=rast_5b73df9c865915 method="average" output=output8a381f2f7cc44b4eae194a33b1839457 --overwrite']
2018-08-15T10:09:00 INFO processInputs end. Commands: ['r.external input="C:\\Users\\rudi\\AppData\\Local\\Temp\\processing_74832af828bb48a9bb0e64fd59d74f24\\1d1e057229aa4ee88a8abcb009b78f41\\hm3.tif" band=1 output="rast_5b73df9c8ac116" --overwrite -o', 'g.region n=-3680453.4502051217 s=-3763597.8879499235 e=-293721.76789138245 w=-478149.5738745922 res=92.0757893076434']
2018-08-15T10:09:00 INFO processCommands end. Commands: ['r.external input="C:\\Users\\rudi\\AppData\\Local\\Temp\\processing_74832af828bb48a9bb0e64fd59d74f24\\1d1e057229aa4ee88a8abcb009b78f41\\hm3.tif" band=1 output="rast_5b73df9c8ac116" --overwrite -o', 'g.region n=-3680453.4502051217 s=-3763597.8879499235 e=-293721.76789138245 w=-478149.5738745922 res=92.0757893076434', 'r.stats.zonal base=rast_5b73df9c8ac116 cover=rast_5b73df9c865915 method="average" output=outputbef87f5fb08c43cd9e1cdd7ad44e5f1b --overwrite']
2018-08-15T10:09:00 INFO processInputs end. Commands: ['r.external input="C:\\Users\\rudi\\AppData\\Local\\Temp\\processing_74832af828bb48a9bb0e64fd59d74f24\\4895f2e889a84f01aa852e7ca0035294\\hm2.tif" band=1 output="rast_5b73df9c8defa7" --overwrite -o', 'g.region n=-3680453.4502051217 s=-3763597.8879499235 e=-293721.76789138245 w=-478149.5738745922 res=92.0757893076434']
2018-08-15T10:09:00 INFO processCommands end. Commands: ['r.external input="C:\\Users\\rudi\\AppData\\Local\\Temp\\processing_74832af828bb48a9bb0e64fd59d74f24\\4895f2e889a84f01aa852e7ca0035294\\hm2.tif" band=1 output="rast_5b73df9c8defa7" --overwrite -o', 'g.region n=-3680453.4502051217 s=-3763597.8879499235 e=-293721.76789138245 w=-478149.5738745922 res=92.0757893076434', 'r.stats.zonal base=rast_5b73df9c8defa7 cover=rast_5b73df9c865915 method="average" output=output349369bf07e14011bfd7611868151ec0 --overwrite']
~~~ [ executed sequentially ] ~~~
2018-08-15T11:46:27 INFO processInputs end. Commands: ['g.proj -c proj4="+proj=aea +lat_1=-24 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs"', 'r.external input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\HabitatModels\\3800-25.tif" band=1 output="rast_5b73f673681b58" --overwrite -o', 'r.external input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\LS90.tif" band=1 output="rast_5b73f673681b59" --overwrite -o', 'g.region n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308 w=-834195.3236270341 res=92.07578930764342']
2018-08-15T11:46:27 INFO processCommands end. Commands: ['g.proj -c proj4="+proj=aea +lat_1=-24 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs"', 'r.external input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\HabitatModels\\3800-25.tif" band=1 output="rast_5b73f673681b58" --overwrite -o', 'r.external input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\LS90.tif" band=1 output="rast_5b73f673681b59" --overwrite -o', 'g.region n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308 w=-834195.3236270341 res=92.07578930764342', 'r.stats.zonal base=rast_5b73f673681b58 cover=rast_5b73f673681b59 method="average" output=output64f55d9cf1ab4a95aa8bedf69f766783 --overwrite']
2018-08-15T11:46:58 INFO processInputs end. Commands: ['g.proj -c proj4="+proj=aea +lat_1=-24 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs"', 'r.external input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\HabitatModels\\3800-25.tif" band=1 output="rast_5b73f692c345310" --overwrite -o', 'r.external input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\LS14.tif" band=1 output="rast_5b73f692c345311" --overwrite -o', 'g.region n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308 w=-834195.3236270341 res=92.07578930764342']
2018-08-15T11:46:58 INFO processCommands end. Commands: ['g.proj -c proj4="+proj=aea +lat_1=-24 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs"', 'r.external input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\HabitatModels\\3800-25.tif" band=1 output="rast_5b73f692c345310" --overwrite -o', 'r.external input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\LS14.tif" band=1 output="rast_5b73f692c345311" --overwrite -o', 'g.region n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308 w=-834195.3236270341 res=92.07578930764342', 'r.stats.zonal base=rast_5b73f692c345310 cover=rast_5b73f692c345311 method="average" output=output1b75839b9455442e95ae87bddfe6c977 --overwrite']
2018-08-15T11:47:30 INFO processInputs end. Commands: ['g.proj -c proj4="+proj=aea +lat_1=-24 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs"', 'r.external input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\HabitatModels\\3800-25.tif" band=1 output="rast_5b73f6b20afa112" --overwrite -o', 'r.external input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\HS90.tif" band=1 output="rast_5b73f6b20afa113" --overwrite -o', 'g.region n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308 w=-834195.3236270341 res=92.07578930764342']
2018-08-15T11:47:30 INFO processCommands end. Commands: ['g.proj -c proj4="+proj=aea +lat_1=-24 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs"', 'r.external input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\HabitatModels\\3800-25.tif" band=1 output="rast_5b73f6b20afa112" --overwrite -o', 'r.external input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\HS90.tif" band=1 output="rast_5b73f6b20afa113" --overwrite -o', 'g.region n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308 w=-834195.3236270341 res=92.07578930764342', 'r.stats.zonal base=rast_5b73f6b20afa112 cover=rast_5b73f6b20afa113 method="average" output=output2c6300f21c524e0da74e65776d6292cf --overwrite']
2018-08-15T11:48:01 INFO processInputs end. Commands: ['g.proj -c proj4="+proj=aea +lat_1=-24 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs"', 'r.external input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\HabitatModels\\3800-25.tif" band=1 output="rast_5b73f6d1a6fa514" --overwrite -o', 'r.external input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\HS14.tif" band=1 output="rast_5b73f6d1a6fa515" --overwrite -o', 'g.region n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308 w=-834195.3236270341 res=92.07578930764342']
2018-08-15T11:48:01 INFO processCommands end. Commands: ['g.proj -c proj4="+proj=aea +lat_1=-24 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs"', 'r.external input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\HabitatModels\\3800-25.tif" band=1 output="rast_5b73f6d1a6fa514" --overwrite -o', 'r.external input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\HS14.tif" band=1 output="rast_5b73f6d1a6fa515" --overwrite -o', 'g.region n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308 w=-834195.3236270341 res=92.07578930764342', 'r.stats.zonal base=rast_5b73f6d1a6fa514 cover=rast_5b73f6d1a6fa515 method="average" output=output744c02ef470a4ddbb2dbaaedf1bda2b0 --overwrite']
On Wed, 15 Aug 2018 at 08:16, Stefan Blumentrath <[Stefan.Blumentrath@nina.no](mailto:Stefan.Blumentrath@nina.no)> wrote:
> Dear Rudi, Nyall,
>
> GRASS is being used on HPC systems for heavily parallelisation. So, in principle, the answer is yes, you can for sure run GRASS algorithms in parallel.
> On Linux, I often run several commands in parallel using xargs. So it works just fine in many cases. GRASS also has some specific python functions for parallel processing. See also:
> [https://grasswiki.osgeo.org/wiki/Parallel_GRASS_jobs](https://grasswiki.osgeo.org/wiki/Parallel_GRASS_jobs)
> [https://grasswiki.osgeo.org/wiki/Parallelizing_Scripts](https://grasswiki.osgeo.org/wiki/Parallelizing_Scripts)
>
> However, if GRASS algorithms can be run in parallel in this particular case depends.
>
> E.g., if the algorithm in question temporarily modifies the computational region, parallel processes can get in the way for each other.
> Also, with SQLite as DB backend writing several vector maps (and attribute tables) in parallel will be a problem (due to SQLite locks).
>
> In addition, if GRASS commands can be executed in parallel in the QGIS Processing framework is probably yet another question, depending on how e.g. QGIS handles data management (locations and mapsets) esp. in more complex workflows / models...
>
> CCing also grass-dev list for more qualified answers...
>
> Cheers
> Stefan
>
> -----Original Message-----
> From: QGIS-Developer <[qgis-developer-bounces@lists.osgeo.org](mailto:qgis-developer-bounces@lists.osgeo.org)> On Behalf Of Nyall Dawson
> Sent: onsdag 15. august 2018 01:10
> To: Rudi von Staden <[rudivs@gmail.com](mailto:rudivs@gmail.com)>
> Cc: qgis-developer <[qgis-developer@lists.osgeo.org](mailto:qgis-developer@lists.osgeo.org)>
> Subject: Re: [QGIS-Developer] Running grass algorithms in threads
>
> On Tue, 14 Aug 2018 at 21:43, Rudi von Staden <[rudivs@gmail.com](mailto:rudivs@gmail.com)> wrote:
> >
> > Hi all,
> >
> > The bottleneck in my script at the moment is the calculation of zonal stats using 'grass7:r.stats.zonal'. I thought I might speed things up by using QgsTask.fromFunction() or QgsProcessingAlgRunnerTask() to run these calculations in parallel. In my tests of both approaches the tasks seem to complete (task.status() == QgsTask.Complete), but the output file is only generated for 1 of 4 parallel tasks (the task that finishes first).
> >
> > I'm assuming this is because grass algorithms are not thread safe? Or am I missing something in my implementation that could make this work?
>
> I strongly suspect that grass algorithms cannot be run in parallel.
> This is why they cannot run in the background in QGIS like the native/GDAL algorithms can. But I'd love for confirmation about this and whether there's any way to make GRASS multi-thread safe.
>
> Because this is grass related (and not QGIS specific) I'd suggest asking on the grass mailing list, and relaying any responses back here.
>
> Nyall
>
> >
> > Thanks,
> > Rudi
> >
> >
> >
> > My code for the QgsTask approach is as below:
> >
> > def getZonal(task, habitatModelFile, cover):
> > tempFile = QgsProcessingUtils.generateTempFilename("output.tif")
> > processing.run("grass7:r.stats.zonal", {
> > 'base':habitatModelFile,
> > 'cover':cover,
> > 'method':5,
> > '-c':False,
> > '-r':False,
> > 'output':tempFile,
> > 'GRASS_REGION_PARAMETER':None,
> > 'GRASS_REGION_CELLSIZE_PARAMETER':0,
> > 'GRASS_RASTER_FORMAT_OPT':'',
> >
> > 'GRASS_RASTER_FORMAT_META':''},context=context,feedback=algFeedback)
> >
> > if task.isCanceled():
> > deleteFile(tempFile)
> > return
> >
> > return tempFile
> >
> > ls90Task = QgsTask.fromFunction('LS90', getZonal,
> > habitatModelFile=hm1, cover=ls90Layer)
> > QgsApplication.taskManager().addTask(ls90Task)
> > feedback.pushInfo("Calculating LS14 mean...") ls14Task =
> > QgsTask.fromFunction('LS14 ', getZonal, habitatModelFile=hm2,
> > cover=ls14Layer)
> > QgsApplication.taskManager().addTask(ls14Task)
> > hs90Task = QgsTask.fromFunction('HS90 ', getZonal,
> > habitatModelFile=hm3, cover=hs90Layer)
> > QgsApplication.taskManager().addTask(hs90Task)
> > hs14Task = QgsTask.fromFunction('HS14 ', getZonal,
> > habitatModelFile=hm4, cover=hs14Layer)
> > QgsApplication.taskManager().addTask(hs14Task)
> >
> > while (len([t for t in [ls90Task.status(), ls14Task.status(), hs90Task.status(),
> > hs14Task.status()] if t in [QgsTask.Running, QgsTask.Queued]]) > 0)
> > and not feedback.isCanceled():
> > sleep(1)
> >
> > if feedback.isCanceled():
> > # some cleanup code (send task.cancel() and wait for tasks to terminate)
> > break
> >
> > ls90Result = ls90Task.returned_values
> > ls14Result = ls14Task.returned_values
> > hs90Result = hs90Task.returned_values # only this file exists
> > hs14Result = hs14Task.returned_values
> >
> >
> > _______________________________________________
> > QGIS-Developer mailing list
> > [QGIS-Developer@lists.osgeo.org](mailto:QGIS-Developer@lists.osgeo.org)
> > List info: [https://lists.osgeo.org/mailman/listinfo/qgis-developer](https://lists.osgeo.org/mailman/listinfo/qgis-developer)
> > Unsubscribe: [https://lists.osgeo.org/mailman/listinfo/qgis-developer](https://lists.osgeo.org/mailman/listinfo/qgis-developer)
> _______________________________________________
> QGIS-Developer mailing list
> [QGIS-Developer@lists.osgeo.org](mailto:QGIS-Developer@lists.osgeo.org)
> List info: [https://lists.osgeo.org/mailman/listinfo/qgis-developer](https://lists.osgeo.org/mailman/listinfo/qgis-developer)
> Unsubscribe: [https://lists.osgeo.org/mailman/listinfo/qgis-developer](https://lists.osgeo.org/mailman/listinfo/qgis-developer)
