[GRASS-dev] measuring cpu time for each module and estimating modules scalability

GRASS GIS GRASS GIS developer
1

Hi all,

just some intellectual thoughts for the weekend (I'm off for a
weekend, so no need to rush with the answer). I started to write some
script [1], which tries to estimate CPU time, needed for each GRASS
module (currently, I have about 4-5 raster modules). Target is to
create suit (rather then simple script), which could estimate how each
process scales (based on number of features or raster cells, which
need to be processed) and put GRASS modules into row. Also estimate
rawly CPU time, which is consumed by each module.

Any thoughts to this?

I started with the dataset from Martin [2], but I thing, I'll switch
over to spearfish, one resolution for now. I did not figure out yet,
how to implement equivalent of unix time [3] program in Python

Thanks

Jachym

[1] https://github.com/jachym/grass-performace
[2] http://geo.fsv.cvut.cz/data/grasswikicz/freegeodatacz/aktualni/
[3] https://github.com/jachym/grass-performace

--
Jachym Cepicky
e-mail: jachym.cepicky gmail com
URL: http://les-ejk.cz
GPG: http://les-ejk.cz/pgp/JachymCepicky.pgp

Give your code freedom with PyWPS - http://pywps.wald.intevation.org

2

Hi Jachym,
here some suggestions from me:
* When using PyGRASS: please create a Module() object before the
benchmark run (run_=False) because PyGRASS will call the GRASS module
to analyse its XML output before any processing is done. This may
influence the runtime and therefore the benchmark significantly for
small data.
* Try to make a sync (man sync) at the end of each run, to avoid
caching of output data in the main memory measurement instead of hard
disk IO performance
* Please consider to integrate your benchmark suite into the GRASS
testing framework GSoC project [1] which already includes runtime
analysis.

Best regards
Soeren

[1] http://trac.osgeo.org/grass/wiki/GSoC/2014/TestingFrameworkForGRASS

2014-08-08 13:00 GMT+02:00 Jachym Cepicky <jachym.cepicky@gmail.com>:

Hi all,

just some intellectual thoughts for the weekend (I'm off for a
weekend, so no need to rush with the answer). I started to write some
script [1], which tries to estimate CPU time, needed for each GRASS
module (currently, I have about 4-5 raster modules). Target is to
create suit (rather then simple script), which could estimate how each
process scales (based on number of features or raster cells, which
need to be processed) and put GRASS modules into row. Also estimate
rawly CPU time, which is consumed by each module.

Any thoughts to this?

I started with the dataset from Martin [2], but I thing, I'll switch
over to spearfish, one resolution for now. I did not figure out yet,
how to implement equivalent of unix time [3] program in Python

Thanks

Jachym

[1] https://github.com/jachym/grass-performace
[2] http://geo.fsv.cvut.cz/data/grasswikicz/freegeodatacz/aktualni/
[3] https://github.com/jachym/grass-performace

--
Jachym Cepicky
e-mail: jachym.cepicky gmail com
URL: http://les-ejk.cz
GPG: http://les-ejk.cz/pgp/JachymCepicky.pgp

Give your code freedom with PyWPS - http://pywps.wald.intevation.org
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

3

On Fri, Aug 8, 2014 at 8:29 AM, Sören Gebbert <soerengebbert@googlemail.com> wrote:

  • Please consider to integrate your benchmark suite into the GRASS
    testing framework GSoC project [1] which already includes runtime
    analysis.

Hi Jachym,

that’s great that you are doing this and that you want to standardize it. I’m not sure what exactly Soeren means by “already includes runtime analysis” because testing framework does not contain anything like that yet (not counting timing included in PyGRASS [2]) but there is certainly an overlap between benchmarking and testing.

First, we plan to include timing into the testing framework, so that when you call a module you get also time as a part of (test) report. Some other things are also interesting, namely profiling, memory check (valgrind), coverage and perhaps some other things from clang/LLVM world [3]. I think that profiling and timing are surely the overlap between testing framework and benchmarking framework/suite. The idea, for testing, currently is that the measurement of module execution would be done in assertModule() function (which executes the module) [4] and the results will be collected for each test.

Second, and I was not much thinking about this yet, benchmark can has a very similar structure to test: preparation, benchmark/test and cleanup. I don’t think that they should be mixed because benchmarks would slow down tests significantly but maybe the system can be the same. On the other hand, there is certainly a large areas in benchmarking API, implementation and report generation which are very different from testing, for example a graph of cell count on X axis and time on Y axis and everything what is needed to get this graph. Also note that some projects actually monitor time of test execution and compare current state to the previous one to warn if there was some slow down (this is applicable for both testing and benchmarking).

Anyway, once the ideas and code are stabilized and relation to testing framework is solved, this is something I would like to see in GRASS core.

2014-08-08 13:00 GMT+02:00 Jachym Cepicky <jachym.cepicky@gmail.com>:

I started with the dataset from Martin, but I thing, I’ll switch
over to spearfish, one resolution for now.

Please, switch to NC SPM, this is the dataset/location which is (should be) used in manual pages now and which is partially used also by tests.

Best,

Vaclav

[1] http://trac.osgeo.org/grass/wiki/GSoC/2014/TestingFrameworkForGRASS
[2] http://trac.osgeo.org/grass/browser/grass/trunk/lib/python/pygrass/modules/interface/module.py#L537
[3] http://trac.osgeo.org/grass/wiki/GSoC/2014/TestingFrameworkForGRASS#AnalyzingmodulerunusingValgrindorothers
[4] http://trac.osgeo.org/grass/browser/grass/trunk/lib/python/gunittest/case.py?rev=61459#L983

2014-08-08 13:00 GMT+02:00 Jachym Cepicky <jachym.cepicky@gmail.com>:

Hi all,

just some intellectual thoughts for the weekend (I’m off for a
weekend, so no need to rush with the answer). I started to write some
script [1], which tries to estimate CPU time, needed for each GRASS
module (currently, I have about 4-5 raster modules). Target is to
create suit (rather then simple script), which could estimate how each
process scales (based on number of features or raster cells, which
need to be processed) and put GRASS modules into row. Also estimate
rawly CPU time, which is consumed by each module.

Any thoughts to this?

I started with the dataset from Martin [2], but I thing, I’ll switch
over to spearfish, one resolution for now. I did not figure out yet,
how to implement equivalent of unix time [3] program in Python

Thanks

Jachym

[1] https://github.com/jachym/grass-performace
[2] http://geo.fsv.cvut.cz/data/grasswikicz/freegeodatacz/aktualni/
[3] https://github.com/jachym/grass-performace

–
Jachym Cepicky
e-mail: jachym.cepicky gmail com
URL: http://les-ejk.cz
GPG: http://les-ejk.cz/pgp/JachymCepicky.pgp

Give your code freedom with PyWPS - http://pywps.wald.intevation.org

grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

4

Hello,

2014-08-08 16:04 GMT+02:00 Vaclav Petras <wenzeslaus@gmail.com>:

On Fri, Aug 8, 2014 at 8:29 AM, Sören Gebbert <soerengebbert@googlemail.com>
wrote:

* Please consider to integrate your benchmark suite into the GRASS
testing framework GSoC project [1] which already includes runtime
analysis.

Hi Jachym,

that's great that you are doing this and that you want to standardize it.
I'm not sure what exactly Soeren means by "already includes runtime
analysis" because testing framework does not contain anything like that yet
(not counting timing included in PyGRASS [2]) but there is certainly an
overlap between benchmarking and testing.

[2] Is what i was referencing with "runtime" not the profiling and
memory checking that we would like to support.

First, we plan to include timing into the testing framework, so that when
you call a module you get also time as a part of (test) report. Some other
things are also interesting, namely profiling, memory check (valgrind),
coverage and perhaps some other things from clang/LLVM world [3]. I think
that profiling and timing are surely the overlap between testing framework
and benchmarking framework/suite. The idea, for testing, currently is that
the measurement of module execution would be done in `assertModule()`
function (which executes the module) [4] and the results will be collected
for each test.

Second, and I was not much thinking about this yet, benchmark can has a very
similar structure to test: preparation, benchmark/test and cleanup. I don't
think that they should be mixed because benchmarks would slow down tests
significantly but maybe the system can be the same. On the other hand, there
is certainly a large areas in benchmarking API, implementation and report
generation which are very different from testing, for example a graph of
cell count on X axis and time on Y axis and everything what is needed to get
this graph. Also note that some projects actually monitor time of test
execution and compare current state to the previous one to warn if there was
some slow down (this is applicable for both testing and benchmarking).

Anyway, once the ideas and code are stabilized and relation to testing
framework is solved, this is something I would like to see in GRASS core.

2014-08-08 13:00 GMT+02:00 Jachym Cepicky <jachym.cepicky@gmail.com>:

I started with the dataset from Martin, but I thing, I'll switch

over to spearfish, one resolution for now.

Please, switch to NC SPM, this is the dataset/location which is (should be)
used in manual pages now and which is partially used also by tests.

Best,
Vaclav

[1] http://trac.osgeo.org/grass/wiki/GSoC/2014/TestingFrameworkForGRASS
[2]
http://trac.osgeo.org/grass/browser/grass/trunk/lib/python/pygrass/modules/interface/module.py#L537
[3]
http://trac.osgeo.org/grass/wiki/GSoC/2014/TestingFrameworkForGRASS#AnalyzingmodulerunusingValgrindorothers
[4]
http://trac.osgeo.org/grass/browser/grass/trunk/lib/python/gunittest/case.py?rev=61459#L983

2014-08-08 13:00 GMT+02:00 Jachym Cepicky <jachym.cepicky@gmail.com>:
> Hi all,
>
> just some intellectual thoughts for the weekend (I'm off for a
> weekend, so no need to rush with the answer). I started to write some
> script [1], which tries to estimate CPU time, needed for each GRASS
> module (currently, I have about 4-5 raster modules). Target is to
> create suit (rather then simple script), which could estimate how each
> process scales (based on number of features or raster cells, which
> need to be processed) and put GRASS modules into row. Also estimate
> rawly CPU time, which is consumed by each module.
>
> Any thoughts to this?
>
> I started with the dataset from Martin [2], but I thing, I'll switch
> over to spearfish, one resolution for now. I did not figure out yet,
> how to implement equivalent of unix time [3] program in Python
>
> Thanks
>
> Jachym
>
> [1] https://github.com/jachym/grass-performace
> [2] http://geo.fsv.cvut.cz/data/grasswikicz/freegeodatacz/aktualni/
> [3] https://github.com/jachym/grass-performace
>
> --
> Jachym Cepicky
> e-mail: jachym.cepicky gmail com
> URL: http://les-ejk.cz
> GPG: http://les-ejk.cz/pgp/JachymCepicky.pgp
>
> Give your code freedom with PyWPS - http://pywps.wald.intevation.org
> _______________________________________________
> grass-dev mailing list
> grass-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev