[GRASS-dev] Testsuite: script for easier use of test framework

GRASS GRASS developer
1

Hi,

I have used information which I found here and there and created a
script to be run locally:

testsuite/test_framework_GRASS_GIS_with_NC.sh

For sure a draft but I needed a script to automatically test if a
GRASS GIS installation is working ok or not.

Since we have the long term confusion if to use "nc_spm_08_grass7" or
"nc_basic_spm_grass7" in test cases, I simply used this trick:

# Preparation: Since we use the full NC dataset, we need to generate
some simplified names as used in NC basic for some test cases
echo "
g.copy raster=basin_50K,basin
g.copy raster=boundary_county_500m,boundary
g.copy raster=landcover_1m,landcover
g.copy raster=geology_30m,geology
g.copy raster=landuse96,landuse
g.copy raster=soilsID,soils
g.copy vector=zipcodes_wake,zipcodes
g.copy vector=schools_wake,schools
" > $GRASSDATA/tmp_rename.sh
$GRASSBIN $GRASSDATA/nc_spm_08_grass7/PERMANENT --exec sh
$GRASSDATA/tmp_rename.sh
rm -f $GRASSDATA/tmp_rename.sh

@Vaclav Petras
Maybe a workaround for the online test system at
http://fatra.cnr.ncsu.edu/grassgistests/summary_report/
?

Best
Markus

2

On Wed, Jan 16, 2019 at 12:00 PM Markus Neteler <neteler@osgeo.org> wrote:

@Vaclav Petras
Maybe a workaround for the online test system at
http://fatra.cnr.ncsu.edu/grassgistests/summary_report/
?

Hi Markus,

the reason I didn’t do that was I was hoping we either decide which one to use or release a new version of full NC SPM with the names from basic (we were quite close to it at one point :-).

If I recall correctly, fatra.cnr.ncsu.edu was originally using the full dataset for maximal variability, then I switched to basic because that was what was applicable also for the piemonte_utm32_wgs84_grass7 dataset (which hides under stdmaps link). This all goes back to the idea of testing framework using different datasets:

  1. full NC SPM (marked nc in lib/python/gunittest/multirunner.py and at fatra),

  2. any Standardized Sample Datasets [1] (marked stdmaps; fatra uses piemonte as an example).

  3. any dataset (marked all; fatra uses an empty XY loc)

This marking is not part of the tests. I never had time to implement that. Part of the mechanism is already in place, although I think there needs to be some clarification on specifying tests which can run in any location (a “universal” test) versus running all tests regardless of their specified location.

I changed the location on fatra to full NC SPM and did the rename as you suggested. I don’t know how many tests does this affect, but in any case tomorrow we will know if I at least set the things correctly.

Best,

Vaclav

[1] https://grasswiki.osgeo.org/wiki/GRASS_GIS_Standardized_Sample_Datasets

3

On Wed, Jan 16, 2019 at 9:53 PM Vaclav Petras <wenzeslaus@gmail.com> wrote:
...
(just quickly while being busy right now)

I changed the location on fatra to full NC SPM and did the rename as you suggested. I don't know how
many tests does this affect, but in any case tomorrow we will know if I at least set the things correctly.

http://fatra.cnr.ncsu.edu/grassgistests/reports_for_date-2019-01-17-08-00/report_for_nc_spm_full_v2alpha_nc/testsuites.html

It seems that it failed. Importantly, the 5 or so maps need to be
_copied_ under new name, not remnamed.

Would you mind to check again?

thanks
Markus

4

On Thu, Jan 17, 2019 at 11:11 AM Markus Neteler <neteler@osgeo.org> wrote:

On Wed, Jan 16, 2019 at 9:53 PM Vaclav Petras <wenzeslaus@gmail.com> wrote:
…
(just quickly while being busy right now)

I changed the location on fatra to full NC SPM and did the rename as you suggested. I don’t know how
many tests does this affect, but in any case tomorrow we will know if I at least set the things correctly.

http://fatra.cnr.ncsu.edu/grassgistests/reports_for_date-2019-01-17-08-00/report_for_nc_spm_full_v2alpha_nc/testsuites.html

It seems that it failed. Importantly, the 5 or so maps need to be
copied under new name, not remnamed.

One of the g.copy failed:

g.copy raster=landuse96,landuse

should be

g.copy raster=landuse96_28m,landuse

Would you mind to check again?

There is an additional issue: the file testsuite/test_framework_GRASS_GIS_with_NC.sh is executed as an test because it is in a directory named testsuite AND it fits pattern *.sh. Putting it into a subdirectory examples is one option how to avoid the problem.

Can you also comment on what are the difference to the example cron job from the documentation [1] (besides the renames)? And whether or not it would be worth it to have just one example, this one or the one in the documentation.

[1] https://grass.osgeo.org/grass76/manuals/libpython/gunittest_running_tests.html#example-bash-script-to-run-be-used-as-a-cron-job

5

On Fri, Jan 18, 2019 at 2:00 AM Vaclav Petras <wenzeslaus@gmail.com> wrote:

On Thu, Jan 17, 2019 at 11:11 AM Markus Neteler <neteler@osgeo.org> wrote:

...

> I changed the location on fatra to full NC SPM and did the rename as you suggested. I don't know how
> many tests does this affect, but in any case tomorrow we will know if I at least set the things correctly.

Thanks!

There is an additional issue: the file testsuite/test_framework_GRASS_GIS_with_NC.sh is executed as an
test because it is in a directory named testsuite AND it fits pattern *.sh. Putting it into a subdirectory examples is one option how to avoid the problem.

sure - moved to subdir.

Can you also comment on what are the difference to the example cron job from the documentation [1] (besides
the renames)? And whether or not it would be worth it to have just one example, this one or the one in the documentation.

[1] https://grass.osgeo.org/grass76/manuals/libpython/gunittest_running_tests.html#example-bash-script-to-run-be-used-as-a-cron-job

Differences:
- script reorganization: I like it much more when definitions are done
on top and not in the script "middle".Moving them to the upper part of
the script + adding a few more vars is one difference.
- now GRASS GIS is compiled with make -j ($MYNPROC - 1), faster for
bigger machines
- compilation can be switched on/off in the definitions
- NC data download added (needed if you test a GRASS GIS installation
on a new machine)
- user can define python or python3 (or whatever, it is a var now)
- publication on Web server can be switched on/off in the definitions

That's more or less it..

Markus

6

On Sun, Jan 20, 2019 at 12:02 PM Markus Neteler <neteler@osgeo.org> wrote:

On Fri, Jan 18, 2019 at 2:00 AM Vaclav Petras <wenzeslaus@gmail.com> wrote:

On Thu, Jan 17, 2019 at 11:11 AM Markus Neteler <neteler@osgeo.org> wrote:
…

I changed the location on fatra to full NC SPM and did the rename as you suggested. I don’t know how
many tests does this affect, but in any case tomorrow we will know if I at least set the things correctly.

I did additional renames (see the test_framework_GRASS_GIS_with_NC.sh script for the full list), but it fixes the same number tests which are now newly broken likely due to differences in basic and full NC SPM (differences other than names). Here is the list of newly broken tests (feel free to investigate):

http://fatra.cnr.ncsu.edu/grassgistests/reports_for_date-2019-01-20-08-00/report_for_nc_spm_full_v2alpha_nc/raster/r.basins.fill/testrbf/index.html

http://fatra.cnr.ncsu.edu/grassgistests/reports_for_date-2019-01-20-08-00/report_for_nc_spm_full_v2alpha_nc/vector/v.surf.rst/test_vsurfrst/index.html

http://fatra.cnr.ncsu.edu/grassgistests/reports_for_date-2019-01-20-08-00/report_for_nc_spm_full_v2alpha_nc/vector/v.univar/v_univar_test/index.html

List of renames:

https://trac.osgeo.org/grass/browser/grass/trunk/testsuite/examples/test_framework_GRASS_GIS_with_NC.sh#L98

Can you also comment on what are the difference to the example cron job from the documentation [1] (besides
the renames)? And whether or not it would be worth it to have just one example, this one or the one in the documentation.

[1] https://grass.osgeo.org/grass76/manuals/libpython/gunittest_running_tests.html#example-bash-script-to-run-be-used-as-a-cron-job

Differences:

  • script reorganization: I like it much more when definitions are done
    on top and not in the script “middle”.Moving them to the upper part of
    the script + adding a few more vars is one difference.
  • now GRASS GIS is compiled with make -j ($MYNPROC - 1), faster for
    bigger machines
  • compilation can be switched on/off in the definitions
  • NC data download added (needed if you test a GRASS GIS installation
    on a new machine)
  • user can define python or python3 (or whatever, it is a var now)
  • publication on Web server can be switched on/off in the definitions

That’s more or less it…

Any reason for the file, i.e. can this be in the documentation? Or do you want a simpler example in doc and a link to the source code with more complex or ready-to-use examples?

Vaclav

7

On Mon, Jan 21, 2019 at 1:48 AM Vaclav Petras <wenzeslaus@gmail.com> wrote:
...

Any reason for the file, i.e. can this be in the documentation?

Not only because...

Or do you want a simpler example in doc and a link to the source code with more complex or ready-to-use examples?

... I need to be able to automatically call the script to test an installation.
We are deploying GRASS GIS in a cloud environment (Openshift based)
and cannot manually copy-paste scripts from the documentation.

Or do I miss something?

Markus

8

On Mon, Jan 21, 2019 at 3:08 PM Markus Neteler <neteler@osgeo.org> wrote:

On Mon, Jan 21, 2019 at 1:48 AM Vaclav Petras <wenzeslaus@gmail.com> wrote:
…

Any reason for the file, i.e. can this be in the documentation?

Not only because…

Or do you want a simpler example in doc and a link to the source code with more complex or ready-to-use examples?

… I need to be able to automatically call the script to test an installation.
We are deploying GRASS GIS in a cloud environment (Openshift based)
and cannot manually copy-paste scripts from the documentation.

OK, this raises the question if I should put the exact script I’m using at fatra.cnr.ncsu.edu to GRASS GIS code? I based the example in doc on that but considered the actual script as too specific (it needs to have the hardcoded paths anyway).

9

On Mon, Jan 21, 2019 at 9:34 PM Vaclav Petras <wenzeslaus@gmail.com> wrote:

On Mon, Jan 21, 2019 at 3:08 PM Markus Neteler <neteler@osgeo.org> wrote:

On Mon, Jan 21, 2019 at 1:48 AM Vaclav Petras <wenzeslaus@gmail.com> wrote:
...
> Any reason for the file, i.e. can this be in the documentation?

Not only because...

> Or do you want a simpler example in doc and a link to the source code with more complex or ready-to-use examples?

... I need to be able to automatically call the script to test an installation.
We are deploying GRASS GIS in a cloud environment (Openshift based)
and cannot manually copy-paste scripts from the documentation.

OK, this raises the question if I should put the exact script I'm using at fatra.cnr.ncsu.edu to GRASS GIS code?

Ah, in order to also run it directly, right?

I based the example in doc on that but considered the actual script as too specific (it needs to have the hardcoded paths anyway).

Perhaps it might be the best to store the hardcoded paths in a
separate config file which is "source"d by the test script?

Markus

10

On Mon, Jan 21, 2019 at 3:59 PM Markus Neteler <neteler@osgeo.org> wrote:

On Mon, Jan 21, 2019 at 9:34 PM Vaclav Petras <wenzeslaus@gmail.com> wrote:

On Mon, Jan 21, 2019 at 3:08 PM Markus Neteler <neteler@osgeo.org> wrote:

On Mon, Jan 21, 2019 at 1:48 AM Vaclav Petras <wenzeslaus@gmail.com> wrote:
…

Any reason for the file, i.e. can this be in the documentation?

Not only because…

Or do you want a simpler example in doc and a link to the source code with more complex or ready-to-use examples?

… I need to be able to automatically call the script to test an installation.
We are deploying GRASS GIS in a cloud environment (Openshift based)
and cannot manually copy-paste scripts from the documentation.

OK, this raises the question if I should put the exact script I’m using at fatra.cnr.ncsu.edu to GRASS GIS code?

Ah, in order to also run it directly, right?

I based the example in doc on that but considered the actual script as too specific (it needs to have the hardcoded paths anyway).

Perhaps it might be the best to store the hardcoded paths in a
separate config file which is "source"d by the test script?

That’s a good idea. That way I we could actually run it directly without storing the specific settings in the source code.

11

On Mon, Jan 21, 2019 at 11:14 PM Vaclav Petras <wenzeslaus@gmail.com> wrote:

On Mon, Jan 21, 2019 at 3:59 PM Markus Neteler <neteler@osgeo.org> wrote:

On Mon, Jan 21, 2019 at 9:34 PM Vaclav Petras <wenzeslaus@gmail.com> wrote:

...

> OK, this raises the question if I should put the exact script I'm using at fatra.cnr.ncsu.edu to GRASS GIS code?

Ah, in order to also run it directly, right?

> I based the example in doc on that but considered the actual script as too specific (it needs to have the hardcoded paths anyway).

Perhaps it might be the best to store the hardcoded paths in a
separate config file which is "source"d by the test script?

That's a good idea. That way I we could actually run it directly without storing the specific settings in the source code.

Changed in r73993.
Please try.

Markus

12

On Mon, Jan 21, 2019 at 5:34 PM Markus Neteler <neteler@osgeo.org> wrote:

On Mon, Jan 21, 2019 at 11:14 PM Vaclav Petras <wenzeslaus@gmail.com> wrote:

On Mon, Jan 21, 2019 at 3:59 PM Markus Neteler <neteler@osgeo.org> wrote:

On Mon, Jan 21, 2019 at 9:34 PM Vaclav Petras <wenzeslaus@gmail.com> wrote:
…

OK, this raises the question if I should put the exact script I’m using at fatra.cnr.ncsu.edu to GRASS GIS code?

Ah, in order to also run it directly, right?

I based the example in doc on that but considered the actual script as too specific (it needs to have the hardcoded paths anyway).

Perhaps it might be the best to store the hardcoded paths in a
separate config file which is "source"d by the test script?

That’s a good idea. That way I we could actually run it directly without storing the specific settings in the source code.

Changed in r73993.
Please try.

How do you suggest to set path the config file to source? I don’t have it in the current directory. Script parameter?

Also, what do you think about keeping all the grass77, grass76, … in the source code? I’m using the following in my script. Would that be understandable in the example?

just get any file which is there (there is only one, but the name differs by version

GRASSBIN=“$GRASSSRC/bin.x86_64-pc-linux-gnu/grass*”

Otherwise, trying to align the two scripts, it seems I need to parametrize every task the script is doing (I want the compilation to keep more cores free, I don’t need the download, untar and rename, and I have different combination and names of locations).

13

On Tue, Jan 22, 2019 at 4:14 AM Vaclav Petras <wenzeslaus@gmail.com> wrote:

On Mon, Jan 21, 2019 at 5:34 PM Markus Neteler <neteler@osgeo.org> wrote:

On Mon, Jan 21, 2019 at 11:14 PM Vaclav Petras <wenzeslaus@gmail.com> wrote:
> On Mon, Jan 21, 2019 at 3:59 PM Markus Neteler <neteler@osgeo.org> wrote:
>> On Mon, Jan 21, 2019 at 9:34 PM Vaclav Petras <wenzeslaus@gmail.com> wrote:
...
>> > OK, this raises the question if I should put the exact script I'm using at fatra.cnr.ncsu.edu to GRASS GIS code?
>>
>> Ah, in order to also run it directly, right?
>>
>> > I based the example in doc on that but considered the actual script as too specific (it needs to have the hardcoded paths anyway).
>>
>> Perhaps it might be the best to store the hardcoded paths in a
>> separate config file which is "source"d by the test script?
>
>
> That's a good idea. That way I we could actually run it directly without storing the specific settings in the source code.

Changed in r73993.
Please try.

How do you suggest to set path the config file to `source`? I don't have it in the current directory. Script parameter?

I have now added support for specifying the conf_file as a parameter in r73996..
Like this the user can have a local conf file and execute the script with that.

Also, what do you think about keeping all the grass77, grass76, ... in the source code? I'm using the following in my script. Would that be understandable in the example?

Sorry, I don't get what you mean.

# just get any file which is there (there is only one, but the name differs by version
GRASSBIN="$GRASSSRC/bin.x86_64-pc-linux-gnu/grass*"

ok, in r73997 the ARCH is now autodetected.

Otherwise, trying to align the two scripts, it seems I need to parametrize every task the script is doing (I want the compilation to keep more cores free,

customizable FREECPU also added to the conf_file.

I don't need the download, untar and rename,

Could be customized as well.

and I have different combination and names of locations).

We should control all this with your local config file - there will be
no way around it.

Markus