[GRASS-dev] v.rast.stats parallel process

GRASS GRASS developer
1

Hi all,

There is any chance that the v.rast.stats will be paralellized?

Best,

lorenzo

2

On 14/06/18 12:24, Lorenzo Bottaccioli wrote:

Hi all,

There is any chance that the v.rast.stats will be paralellized?

You mean to treat several rasters at the same time ?

It shouldn't be too complicated to write a wrapper script for this. I would imagine the biggest problem to be the handling of the access to the attribute database file. Not sure that you can have concurrent access to this file by several parallel processes. If this is an issue, we would probably have to add the option to output v.rast.stats results to text files and then load content of these files into at the attribute table of the vector map at the end...

Moritz

3

Yes, access to attributes would be a challenge / bottleneck for parallelisation, esp. if you have many areas in your vector map. Note that not only SQLite but also DBF is still supported (which puts some limitations)...

Please note also: https://trac.osgeo.org/grass/ticket/3523 for some suggestions for speeding up v.rast.stats (without parallelisation).
Once multiple raster map input is supported, statistics could be computed in parallel for each raster map and then finally uploaded to the attribute table in one single process...
Yet, the region adjustment performed by v.rast.stats would then have to be handled differently from current solution...

I would say it is worth a ticket...

If you want a text file, you could use r.univar directly, no?

Cheers
Stefan

-----Original Message-----
From: grass-dev <grass-dev-bounces@lists.osgeo.org> On Behalf Of Moritz Lennert
Sent: torsdag 14. juni 2018 13:05
To: Lorenzo Bottaccioli <lorenzo.bottaccioli@gmail.com>; GRASS developers list <grass-dev@lists.osgeo.org>
Subject: Re: [GRASS-dev] v.rast.stats parallel process

On 14/06/18 12:24, Lorenzo Bottaccioli wrote:

Hi all,

There is any chance that the v.rast.stats will be paralellized?

You mean to treat several rasters at the same time ?

It shouldn't be too complicated to write a wrapper script for this. I would imagine the biggest problem to be the handling of the access to the attribute database file. Not sure that you can have concurrent access to this file by several parallel processes. If this is an issue, we would probably have to add the option to output v.rast.stats results to text files and then load content of these files into at the attribute table of the vector map at the end...

Moritz
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

4

Hi Moritz,

No I mean to process in parallel the stats over many vector entities for example areas. Immagine a large raster with many polygons that want to know the mean or max value in the area.

Ciao,

Lorenzo

···

2018-06-14 13:20 GMT+02:00 Stefan Blumentrath <Stefan.Blumentrath@nina.no>:

Yes, access to attributes would be a challenge / bottleneck for parallelisation, esp. if you have many areas in your vector map. Note that not only SQLite but also DBF is still supported (which puts some limitations)…

Please note also: https://trac.osgeo.org/grass/ticket/3523 for some suggestions for speeding up v.rast.stats (without parallelisation).
Once multiple raster map input is supported, statistics could be computed in parallel for each raster map and then finally uploaded to the attribute table in one single process…
Yet, the region adjustment performed by v.rast.stats would then have to be handled differently from current solution…

I would say it is worth a ticket…

If you want a text file, you could use r.univar directly, no?

Cheers
Stefan

-----Original Message-----
From: grass-dev <grass-dev-bounces@lists.osgeo.org> On Behalf Of Moritz Lennert
Sent: torsdag 14. juni 2018 13:05
To: Lorenzo Bottaccioli <lorenzo.bottaccioli@gmail.com>; GRASS developers list <grass-dev@lists.osgeo.org>
Subject: Re: [GRASS-dev] v.rast.stats parallel process

On 14/06/18 12:24, Lorenzo Bottaccioli wrote:

Hi all,

There is any chance that the v.rast.stats will be paralellized?

You mean to treat several rasters at the same time ?

It shouldn’t be too complicated to write a wrapper script for this. I would imagine the biggest problem to be the handling of the access to the attribute database file. Not sure that you can have concurrent access to this file by several parallel processes. If this is an issue, we would probably have to add the option to output v.rast.stats results to text files and then load content of these files into at the attribute table of the vector map at the end…

Moritz

grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

5

On 14/06/18 16:31, Lorenzo Bottaccioli wrote:

Hi Moritz,

No I mean to process in parallel the stats over many vector entities for example areas. Immagine a large raster with many polygons that want to know the mean or max value in the area.

Ok. As the module is a scripted fronted to r.univar, the question is more for r.univar then. Parallizing that will probably be a bit more complicated.

Moritz

Ciao,

Lorenzo

2018-06-14 13:20 GMT+02:00 Stefan Blumentrath <Stefan.Blumentrath@nina.no <mailto:Stefan.Blumentrath@nina.no>>:

    Yes, access to attributes would be a challenge / bottleneck for
    parallelisation, esp. if you have many areas in your vector map.
    Note that not only SQLite but also DBF is still supported (which
    puts some limitations)...

    Please note also: https://trac.osgeo.org/grass/ticket/3523
    <https://trac.osgeo.org/grass/ticket/3523&gt; for some suggestions for
    speeding up v.rast.stats (without parallelisation).
    Once multiple raster map input is supported, statistics could be
    computed in parallel for each raster map and then finally uploaded
    to the attribute table in one single process...
    Yet, the region adjustment performed by v.rast.stats would then have
    to be handled differently from current solution...

    I would say it is worth a ticket...

    If you want a text file, you could use r.univar directly, no?

    Cheers
    Stefan

    -----Original Message-----
    From: grass-dev <grass-dev-bounces@lists.osgeo.org
    <mailto:grass-dev-bounces@lists.osgeo.org>> On Behalf Of Moritz Lennert
    Sent: torsdag 14. juni 2018 13:05
    To: Lorenzo Bottaccioli <lorenzo.bottaccioli@gmail.com
    <mailto:lorenzo.bottaccioli@gmail.com>>; GRASS developers list
    <grass-dev@lists.osgeo.org <mailto:grass-dev@lists.osgeo.org>>
    Subject: Re: [GRASS-dev] v.rast.stats parallel process

    On 14/06/18 12:24, Lorenzo Bottaccioli wrote:
     > Hi all,
     >
     > There is any chance that the v.rast.stats will be paralellized?

    You mean to treat several rasters at the same time ?

    It shouldn't be too complicated to write a wrapper script for this.
    I would imagine the biggest problem to be the handling of the access
    to the attribute database file. Not sure that you can have
    concurrent access to this file by several parallel processes. If
    this is an issue, we would probably have to add the option to output
    v.rast.stats results to text files and then load content of these
    files into at the attribute table of the vector map at the end...

    Moritz
    _______________________________________________
    grass-dev mailing list
    grass-dev@lists.osgeo.org <mailto:grass-dev@lists.osgeo.org>
    https://lists.osgeo.org/mailman/listinfo/grass-dev
    <https://lists.osgeo.org/mailman/listinfo/grass-dev&gt;