[GRASS-dev] Weekly report 10: Testing framework for GRASS GIS

  1. What did you get done this week?

I significantly improved the test file page which is the central page of the report in the sense that there you can get all the details. I also added details about successful and failed (individual) tests to main index page (the one with links to all test files). Additionally, I created different set of pages which sorts the test files from the point of view of testsuite directories which might be the primary way of browsing tests since it is aligned with how tests are included in the source code. The page for a given testsuite also contains approximate information about authors of tests and authors of tested code.

Each test file has now a key-value file which contains information about success, number of tests, and other information. These files can and are collected to obtain summaries. The key-value format was selected mainly because of its usage in GRASS GIS (otherwise, for example Python documentation for inputs and outputs recommends JSON [1]).

I believe that I designed report classes in the way that they can be extended but already I can see some room for refactoring or at least for better organization. This might be a topic for week 13.

I did not created any XML output. The situation seems unclear and a lot of transformations are done to get XML working with different tools. On the other hand, with some tools even trivial XML with numbers (test statistics) should work and this should be easy to create from key-value files or directly. Also there is a unittest-xml-reporting package [2] for Python unittest which should create XML in the style of Ant/JUnit. In theory this should work directly with gunittest. I was not considering database output much but I think that it is not needed now and it can be created from key-value files if necessary.

A also did not focus on CSS of the HTML reports. They are readable but that’s all. Some CSS might be added in the future.

There is currently 20 successfully running test files and 5 failing test files, 9 successfully running testsuites and 4 failing testsuites, and 686 successfully running tests and 7 failing tests (533 of the tests are from PyGRASS module test).

I also added some tests for r.mapcalc rand() function and new behavior with seed in r61355.

See the project wiki page for screenshots and more detailed description of the test report or obtain one yourself running:

in the grass souce code directory, in grass session

python -m grass.gunittest.main gisdbasepath nc_smp_location_name nc

open file “testreport/index.html” or “testreport/testsuites.html”

  1. What do you plan on doing next week?

I plan to rewrite some of the existing tests using gunittest or write new ones. I already started to write something for r.slope.aspect and r.mapcalc.

If time allows, I would like to add some key-value summary files to reports, include some test metadata such as GRASS location or architecture, and also introduce more options to report generation (currently, it cannot be configured).

Additionally, It would be nice to have all path in files shortened so that they don’t include dist... or full path to the file and also it would be nice to have an example of running the tests automatically in some trivial form.

  1. Are you blocked on anything?

I think that some CSS and JavaScript can improve the reports significantly but I did not included any nice design to HTML pages because it would cost me too much time (and it was not part of the GSoC plan anyway).

Wiki with screenshots:
http://trac.osgeo.org/grass/wiki/GSoC/2014/TestingFrameworkForGRASS#Week10

[1] https://docs.python.org/2/tutorial/inputoutput.html#saving-structured-data-with-json
[2] https://github.com/xmlrunner/unittest-xml-reporting