[GRASS-dev] Installing own Python script

GRASS GRASS developer
1

Hi,

I'm beginning to script GRASS using Python. Turns out it is very easy
and convivial, and in a very limited amount of time I was able to put
together a little extension to bring the main mathematical morphology
operators to GRASS.

The problem I got, however, is how to install that script into
path/to/grass_src/scripts. In that director, I created a folder named
after my extension (r.mm). It contains:
- my script, r.mm.py
- its doc, r.mm.html, generated by the command python r.mm.py
--html-description, and then completed by hand
- A Makefile

The Makefile is a simple file containing the following lines:

MODULE_TOPDIR = ../..

PGM = r.mm

include $(MODULE_TOPDIR)/include/Make/Script.make

default: script

I've been able to compile the thing once, using make. But I can't run
the make install:

roudierp@A208_RoudierP:/usr/local/src/grass7-svn/grass_trunk/scripts/r.mm> make
/usr/bin/install -c r.mm.py
/usr/local/src/grass7-svn/grass_trunk/dist.x86_64-unknown-linux-gnu/scripts/r.mm
if [ "/usr/local/src/grass7-svn/grass_trunk/dist.x86_64-unknown-linux-gnu/scripts/r.mm"
!= "" ] ; then GISRC=/usr/local/src/grass7-svn/grass_trunk/dist.x86_64-unknown-linux-gnu/demolocation/.grassrc70
GISBASE=/usr/local/src/grass7-svn/grass_trunk/dist.x86_64-unknown-linux-gnu
PATH="/usr/local/src/grass7-svn/grass_trunk/dist.x86_64-unknown-linux-gnu/bin:$PATH"
PYTHONPATH="/usr/local/src/grass7-svn/grass_trunk/dist.x86_64-unknown-linux-gnu/etc/python:$PYTHONPATH"
LD_LIBRARY_PATH="/usr/local/src/grass7-svn/grass_trunk/dist.x86_64-unknown-linux-gnu/bin:/usr/local/src/grass7-svn/grass_trunk/dist.x86_64-unknown-linux-gnu/lib:/usr/local/src/grass7-svn/grass_trunk/dist.x86_64-unknown-linux-gnu/lib:"
LC_ALL=C /usr/local/src/grass7-svn/grass_trunk/dist.x86_64-unknown-linux-gnu/scripts/r.mm
--html-description < /dev/null | grep -v '</body>\|</html>' >
r.mm.tmp.html ; fi
python /usr/local/src/grass7-svn/grass_trunk/dist.x86_64-unknown-linux-gnu/tools/mkhtml.py
r.mm > /usr/local/src/grass7-svn/grass_trunk/dist.x86_64-unknown-linux-gnu/docs/html/r.mm.html
VERSION_NUMBER=7.0.svn
/usr/local/src/grass7-svn/grass_trunk/dist.x86_64-unknown-linux-gnu/tools/g.html2man.py
/usr/local/src/grass7-svn/grass_trunk/dist.x86_64-unknown-linux-gnu/docs/html/r.mm.html
/usr/local/src/grass7-svn/grass_trunk/dist.x86_64-unknown-linux-gnu/man/man1/r.mm.1
GISRC=/usr/local/src/grass7-svn/grass_trunk/dist.x86_64-unknown-linux-gnu/demolocation/.grassrc70
GISBASE=/usr/local/src/grass7-svn/grass_trunk/dist.x86_64-unknown-linux-gnu
PATH="/usr/local/src/grass7-svn/grass_trunk/dist.x86_64-unknown-linux-gnu/bin:$PATH"
PYTHONPATH="/usr/local/src/grass7-svn/grass_trunk/dist.x86_64-unknown-linux-gnu/etc/python:$PYTHONPATH"
LD_LIBRARY_PATH="/usr/local/src/grass7-svn/grass_trunk/dist.x86_64-unknown-linux-gnu/bin:/usr/local/src/grass7-svn/grass_trunk/dist.x86_64-unknown-linux-gnu/lib:/usr/local/src/grass7-svn/grass_trunk/dist.x86_64-unknown-linux-gnu/lib:"
LC_ALL=C g.parser -t r.mm.py | sed s/\"/\\\\\"/g | sed 's/.*/_("&")/'

/usr/local/src/grass7-svn/grass_trunk/locale/scriptstrings/r.mm_to_translate.c

rm r.mm.tmp.html
roudierp@A208_RoudierP:/usr/local/src/grass7-svn/grass_trunk/scripts/r.mm>
make install
make: *** No rule to make target `install'. Stop.

The wiki page [0] is great, but unfortunately does not provides much
information on that specific point (yet). I'd be more than happy to
contribute it.. as soon as I'll get my head around it!

Any help would be appreciated!

Cheers,

Pierre

[0] http://grass.osgeo.org/wiki/GRASS_and_Python

--
Scientist
Landcare Research, New Zealand

2

Pierre Roudier wrote:

I'm beginning to script GRASS using Python. Turns out it is very easy
and convivial, and in a very limited amount of time I was able to put
together a little extension to bring the main mathematical morphology
operators to GRASS.

The problem I got, however, is how to install that script into
path/to/grass_src/scripts. In that director, I created a folder named
after my extension (r.mm). It contains:
- my script, r.mm.py
- its doc, r.mm.html, generated by the command python r.mm.py
--html-description, and then completed by hand
- A Makefile

The r.mm.html file should not contain the parts which are generated by
"r.mm.py --html-description". The build system does this automatically
if the .html file doesn't contain an "<html>" tag. This ensures that
the .html file is kept up to date if options are added, removed or
changed.

I've been able to compile the thing once, using make. But I can't run
the make install:

roudierp@A208_RoudierP:/usr/local/src/grass7-svn/grass_trunk/scripts/r.mm>
make install
make: *** No rule to make target `install'. Stop.

There is no "install" target for scripts, only for (compiled) modules.
You would have the same issue for any of the existing scripts.

Normally, GRASS is installed as a whole by running "make install" from
the top-level directory, which copies the staging directory
(dist.<arch>) to the final location.

The per-module "install" targets are for developers wishing to install
individual modules after modifications. As GRASS can be run from the
staging directory without being installed, this isn't often required.

If this feature is desired for scripts, it should just be a case of
adding a slight modification of r43657 to Script.make (change "bin" to
"scripts" in the first command).

--
Glynn Clements <glynn@gclements.plus.com>

3

Hi Glynn, and thanks very much for taking the time to answer my newbie
questions,

The r.mm.html file should not contain the parts which are generated by
"r.mm.py --html-description". The build system does this automatically
if the .html file doesn't contain an "<html>" tag. This ensures that
the .html file is kept up to date if options are added, removed or
changed.

OK, so I understand the HTML file would be generated automatically at
build time (during the "make" from the top-level dir). Does that mean
that I can't add more descriptions/references/notes to the HTML file
manually?

The per-module "install" targets are for developers wishing to install
individual modules after modifications. As GRASS can be run from the
staging directory without being installed, this isn't often required.

If this feature is desired for scripts, it should just be a case of
adding a slight modification of r43657 to Script.make (change "bin" to
"scripts" in the first command).

What I would like is:
1 - Make my r.mm script available from the GRASS prompt, the same way
other Python scripts like r.grow or r.buffer are available, so I can
test it,
2 - Most importantly, I would like to access my r.mm module from other
Python scripts using eg grass.run_command('r.mm', input = input, ...,
flags = flags, quiet = quiet)

Does that mean I should recompile GRASS entirely after each
modification made to my r.mm module, or is the per-module install
option just doing that (sorry I'm not sure I got it right)?

Many thanks,

Pierre

--
Scientist
Landcare Research, New Zealand

4

Pierre Roudier wrote:

> The r.mm.html file should not contain the parts which are generated by
> "r.mm.py --html-description". The build system does this automatically
> if the .html file doesn't contain an "<html>" tag. This ensures that
> the .html file is kept up to date if options are added, removed or
> changed.

OK, so I understand the HTML file would be generated automatically at
build time (during the "make" from the top-level dir). Does that mean
that I can't add more descriptions/references/notes to the HTML file
manually?

The build system merges the --html-description output with the
<module>.html file (unless the <module>.html file contains an <html>
tag, in which case it is used verbatim; this is intended for modules
which don't recognise the --html-description switch, e.g. g.parser).

Look at the <module>.html files for existing scripts for reference.
They start with the DESCRIPTION section.

> The per-module "install" targets are for developers wishing to install
> individual modules after modifications. As GRASS can be run from the
> staging directory without being installed, this isn't often required.
>
> If this feature is desired for scripts, it should just be a case of
> adding a slight modification of r43657 to Script.make (change "bin" to
> "scripts" in the first command).

What I would like is:
1 - Make my r.mm script available from the GRASS prompt, the same way
other Python scripts like r.grow or r.buffer are available, so I can
test it,
2 - Most importantly, I would like to access my r.mm module from other
Python scripts using eg grass.run_command('r.mm', input = input, ...,
flags = flags, quiet = quiet)

Does that mean I should recompile GRASS entirely after each
modification made to my r.mm module, or is the per-module install
option just doing that (sorry I'm not sure I got it right)?

If you're actively developing GRASS, you wouldn't normally install it.
Just run it using the bin.<arch>/grass70 script. This will set the
environment variables (GISBASE, PATH, etc) to use the version in the
dist.<arch> staging directory.

Running "make" in a module/script/library/etc directory places the
resulting files in the staging directory. If you're running GRASS from
the staging directory, subsequent commands will used the updated
files.

Running "make install" from the top level just copies the whole of
dist.<arch> to the installation directory (e.g. /usr/local/grass70)
and bin.<arch>/grass70 to the bin directory (e.g. /usr/local/bin), and
fixes any embedded paths in scripts and configuration files.

The per-module "install" target (which doesn't exist for scripts at
present) copies a single module (along with its manual page in HTML
and man formats) from the dist.<arch> directory to the installation
directory.

AFAICT, this is only useful if you're pushing your changes into the
installed version so frequently that having to do a full "make install"
from the top level each time would actually be a problem.

--
Glynn Clements <glynn@gclements.plus.com>

5

Hi Glynn,

The build system merges the --html-description output with the
<module>.html file (unless the <module>.html file contains an <html>
tag, in which case it is used verbatim; this is intended for modules
which don't recognise the --html-description switch, e.g. g.parser).

Look at the <module>.html files for existing scripts for reference.
They start with the DESCRIPTION section.

Indeed, my bad - I did looked into it, but did not realized that.

If you're actively developing GRASS, you wouldn't normally install it.
Just run it using the bin.<arch>/grass70 script. This will set the
environment variables (GISBASE, PATH, etc) to use the version in the
dist.<arch> staging directory.

Running "make" in a module/script/library/etc directory places the
resulting files in the staging directory. If you're running GRASS from
the staging directory, subsequent commands will used the updated
files.

Running "make install" from the top level just copies the whole of
dist.<arch> to the installation directory (e.g. /usr/local/grass70)
and bin.<arch>/grass70 to the bin directory (e.g. /usr/local/bin), and
fixes any embedded paths in scripts and configuration files.

Thanks Glynn, this is all working!

The per-module "install" target (which doesn't exist for scripts at
present) copies a single module (along with its manual page in HTML
and man formats) from the dist.<arch> directory to the installation
directory.

AFAICT, this is only useful if you're pushing your changes into the
installed version so frequently that having to do a full "make install"
from the top level each time would actually be a problem.

Which could be interesting when developing that module/script? But for
the time being, I'll use the "make on script directory"+"make install
in the top dir" approach.

Thanks heaps. Should that be on a wiki page somewhere?

Pierre

--
Scientist
Landcare Research, New Zealand

6

On Wed, Jun 1, 2011 at 12:10 AM, Pierre Roudier
<pierre.roudier@gmail.com> wrote:
...

Thanks heaps. Should that be on a wiki page somewhere?

Please expand this page:
http://grass.osgeo.org/wiki/GRASS_and_Python

thanks
Markus

7

http://grass.osgeo.org/wiki/GRASS_and_Python#Parsing_the_options_and_flags
http://grass.osgeo.org/wiki/GRASS_and_Python#Testing_and_installing_Python_extensions

Feedback/corrections welcome.

Thanks to Glynn for his great explanations!

Pierre

2011/6/1 Markus Neteler <neteler@osgeo.org>:

On Wed, Jun 1, 2011 at 12:10 AM, Pierre Roudier
<pierre.roudier@gmail.com> wrote:
...

Thanks heaps. Should that be on a wiki page somewhere?

Please expand this page:
http://grass.osgeo.org/wiki/GRASS_and_Python

thanks
Markus

--
Scientist
Landcare Research, New Zealand

8

Pierre Roudier wrote:

> The per-module "install" target (which doesn't exist for scripts at
> present) copies a single module (along with its manual page in HTML
> and man formats) from the dist.<arch> directory to the installation
> directory.
>
> AFAICT, this is only useful if you're pushing your changes into the
> installed version so frequently that having to do a full "make install"
> from the top level each time would actually be a problem.

Which could be interesting when developing that module/script?

If you're developing, you should be running GRASS from the staging
directory rather than using an installed version.

Apart from convenience, leaving the installed version alone means that
you can compare the behaviour of the modified version to the installed
version.

FWIW, the only time I run "make install" is if I've been working on
the Makefiles and I want to check that "make install" still works.

--
Glynn Clements <glynn@gclements.plus.com>

9

Hi,

2011/6/1 Glynn Clements <glynn@gclements.plus.com>:

> The per-module "install" target (which doesn't exist for scripts at
> present) copies a single module (along with its manual page in HTML
> and man formats) from the dist.<arch> directory to the installation
> directory.
>
> AFAICT, this is only useful if you're pushing your changes into the
> installed version so frequently that having to do a full "make install"
> from the top level each time would actually be a problem.

Which could be interesting when developing that module/script?

If you're developing, you should be running GRASS from the staging
directory rather than using an installed version.

Apart from convenience, leaving the installed version alone means that
you can compare the behaviour of the modified version to the installed
version.

FWIW, the only time I run "make install" is if I've been working on
the Makefiles and I want to check that "make install" still works.

I think that there should the per-script "install" target. At least
for g.extension. Currently, g.extension fails with error.

Installing 'r.threshold'...
make: *** No rule to make target `install'. Stop.
WARNING: Installation failed, sorry. Please check above error messages.

Martin

--
Martin Landa <landa.martin gmail.com> * http://geo.fsv.cvut.cz/~landa

10

2011/6/13 Martin Landa <landa.martin@gmail.com>:

Hi,

2011/6/1 Glynn Clements <glynn@gclements.plus.com>:

> The per-module "install" target (which doesn't exist for scripts at
> present) copies a single module (along with its manual page in HTML
> and man formats) from the dist.<arch> directory to the installation
> directory.
>
> AFAICT, this is only useful if you're pushing your changes into the
> installed version so frequently that having to do a full "make install"
> from the top level each time would actually be a problem.

Which could be interesting when developing that module/script?

If you're developing, you should be running GRASS from the staging
directory rather than using an installed version.

Apart from convenience, leaving the installed version alone means that
you can compare the behaviour of the modified version to the installed
version.

FWIW, the only time I run "make install" is if I've been working on
the Makefiles and I want to check that "make install" still works.

I think that there should the per-script "install" target. At least
for g.extension. Currently, g.extension fails with error.

Installing 'r.threshold'...
make: *** No rule to make target `install'. Stop.
WARNING: Installation failed, sorry. Please check above error messages.

I have the same problem, but the command it was compiled and installed....

Martin

--
regards
Luca

http://gis.cri.fmach.it/delucchi/
www.lucadelu.org

11

Hi,

2011/6/13 Martin Landa <landa.martin@gmail.com>:

[...]

I think that there should the per-script "install" target. At least
for g.extension. Currently, g.extension fails with error.

done in r46687 (review needed).

Installing 'r.threshold'...
make: *** No rule to make target `install'. Stop.
WARNING: Installation failed, sorry. Please check above error messages.

After updating g.extension in G7, it will also install scripts

Installing 'r.threshold'...
/usr/bin/install -c
/home/martin/grassdata/nc_spm_08/landa/.tmp/pierre/17412.0/r.threshold/scripts/r.threshold
/home/martin/.grass7/addons//scripts/
/usr/bin/install -c -m 644
/home/martin/grassdata/nc_spm_08/landa/.tmp/pierre/17412.0/r.threshold/docs/html/r.threshold.html
/home/martin/.grass7/addons//docs/html/
/usr/bin/install -c -m 644
/home/martin/grassdata/nc_spm_08/landa/.tmp/pierre/17412.0/r.threshold/man/man1/r.threshold.1
/home/martin/.grass7/addons//man/man1/
Installation of 'r.threshold' successfully finished.

Martin

--
Martin Landa <landa.martin gmail.com> * http://geo.fsv.cvut.cz/~landa

12

Martin wrote:

done in r46687 (review needed).

n.b. r46690 code still refers to .grass7 in grass6
[please don't backport to 6.4 until code is tested in 6.5 for some
time, even if it seems harmless. otherwise there is no quality
control, the laws of unintended consequences bite us, and what's
the point of maintaining a 6.5 testing branch at all? tx]

addon environ variable can have multiples, you need to spit on
the first ":".

the shell script version in 6.x already installs to the addon dir
as part of g.extension.sh, with sudo,su options to go to the
install dir upon request. should we stay with that and/or is a specific install target needed? (why is a python clone needed
there? to help with menu integration? is there a bug in wingrass
I don't know about?)

Hamish

13

Hamish wrote:

addon environ variable can have multiples, you need to spit
on the first ":".

ugg, C:\ probably doesn't like that.

H

14

On Jun 13, 2011, at 4:17 PM, Hamish wrote:

Martin wrote:

done in r46687 (review needed).

n.b. r46690 code still refers to .grass7 in grass6
[please don't backport to 6.4 until code is tested in 6.5 for some
time, even if it seems harmless. otherwise there is no quality
control, the laws of unintended consequences bite us, and what's
the point of maintaining a 6.5 testing branch at all? tx]

just a note regarding your testing comment
- we discussed the need for testing before backporting to 6.4 with Martin,
so the main issue is to stick with it and make sure it happens.
Hopefully we don't need any official procedure for guarding the stability of
what should be a stable release,

Helena

addon environ variable can have multiples, you need to spit on
the first ":".

the shell script version in 6.x already installs to the addon dir
as part of g.extension.sh, with sudo,su options to go to the
install dir upon request. should we stay with that and/or is a specific install target needed? (why is a python clone needed
there? to help with menu integration? is there a bug in wingrass
I don't know about?)

Hamish

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

15

Helena wrote:

Hopefully we don't need any official procedure for guarding
the stability of what should be a stable release,

I don't think we do, my main motivator in this is remembering two releases
where two of my last minute changes broke the release for months. In one I
"just" broke ps.map, the other I broken the main start-up for first time
users* and spent six months responding to bug reports with "mea culpa,
please try building the latest from svn".
[*] IIRC a G_fatal_error() in a sub-sub-function call, before the module
was init'd. Very subtle bug, only time & testing by new users found it.

and then there was the time in the dev branch one of my shell scripts
did rm -rf "$UNSET_TMP_FILE"* and deleted someone's home dir... and the
time that using the stable branch for my production r.sun runs saved
a month's worth of computations as the dev branch had a new+unnoticed bug
in it.. well, I could go on but the point is that now as a result I am
uber-cautious about what goes into the stable branch, that's all. (Some
push-back from me being too cautious sometimes is healthy of course :slight_smile:

Hamish

16

Hi,

2011/6/13 Hamish <hamish_b@yahoo.com>:

n.b. r46690 code still refers to .grass7 in grass6
[please don't backport to 6.4 until code is tested in 6.5 for some
time, even if it seems harmless. otherwise there is no quality
control, the laws of unintended consequences bite us, and what's
the point of maintaining a 6.5 testing branch at all? tx]

we are in the first half of release cycle (counting that 6.4.2 could
be released in November/December) so I think that we don't be so much
pedandic at this stage. 2-3 four months before release we could froze
relbr64, focus on testing, bugfixing. This make sense for me.

the shell script version in 6.x already installs to the addon dir
as part of g.extension.sh, with sudo,su options to go to the
install dir upon request. should we stay with that and/or is a specific install target needed? (why is a python clone needed
there? to help with menu integration? is there a bug in wingrass
I don't know about?)

Well, it's not clone at all! It's has some extra functionality
compared bash version of g.extension. Moreover it's actively developed
and maintained. This functionality is required by wxGUI, in other
words wxGUI uses pythonized g.extension. I am interested to develop
python version of g.extension, not the bash script. There is no chance
to replace bash script by pythonized clone in G6 mainly because of
your attitude. It's closed story for me. So I decided to include at
least pythonized g.extension to G6 which is required by wxGUI.

Martin

--
Martin Landa <landa.martin gmail.com> * http://geo.fsv.cvut.cz/~landa

17

2011/6/13 Martin Landa <landa.martin@gmail.com>:

Hi,

Hi

Installing 'r.threshold'...
/usr/bin/install -c
/home/martin/grassdata/nc_spm_08/landa/.tmp/pierre/17412.0/r.threshold/scripts/r.threshold
/home/martin/.grass7/addons//scripts/
/usr/bin/install -c -m 644
/home/martin/grassdata/nc_spm_08/landa/.tmp/pierre/17412.0/r.threshold/docs/html/r.threshold.html
/home/martin/.grass7/addons//docs/html/
/usr/bin/install -c -m 644
/home/martin/grassdata/nc_spm_08/landa/.tmp/pierre/17412.0/r.threshold/man/man1/r.threshold.1
/home/martin/.grass7/addons//man/man1/
Installation of 'r.threshold' successfully finished.

I tried it, it works but I have this message

/bin/sh: cannot create
/usr/local/grass-7.0.svn/locale/scriptstrings/v.median_to_translate.c:
Directory nonexistent
sed: couldn't flush stdout: Broken pipe
make: [/usr/local/grass-7.0.svn/locale/scriptstrings/v.median_to_translate.c]
Error 2 (ignored)
rm v.median.tmp.html
Installing 'v.median'...

Martin

--
cheers
Luca

http://gis.cri.fmach.it/delucchi/
www.lucadelu.org

18

Hi,

2011/6/14 Hamish <hamish_b@yahoo.com>:

I don't think we do, my main motivator in this is remembering two releases
where two of my last minute changes broke the release for months. In one I

right. On the other hand you need to bear in mind at which point of
release cycle you are. In the case that we would like to release 6.4.2
in Nov/Dec we are slightly in the middle. I don't think that we need
to be so much pedantic all time. 2-3 mounts before release we should
focus on testing, bugfixing, etc. - i.e. before RC1. Anyway as Helena
mentioned we need solid testing procedure.

Martin

--
Martin Landa <landa.martin gmail.com> * http://geo.fsv.cvut.cz/~landa

19

Hi,

2011/6/14 Luca Delucchi <lucadeluge@gmail.com>:

I tried it, it works but I have this message

/bin/sh: cannot create
/usr/local/grass-7.0.svn/locale/scriptstrings/v.median_to_translate.c:
Directory nonexistent
sed: couldn't flush stdout: Broken pipe
make: [/usr/local/grass-7.0.svn/locale/scriptstrings/v.median_to_translate.c]
Error 2 (ignored)
rm v.median.tmp.html
Installing 'v.median'...

I am not sure how to solve this problem. We could

* add `scriptstrings` to $ARCH_DISTDIR/locale
* check if `scriptstrings` exists, if not just skip it

In any case strings will be not translatable for installed script.

Martin

--
Martin Landa <landa.martin gmail.com> * http://geo.fsv.cvut.cz/~landa

20

Martin wrote:

right. On the other hand you need to bear in mind at which
point of release cycle you are. In the case that we would like to
release 6.4.2 in Nov/Dec we are slightly in the middle. I don't think
that we need to be so much pedantic all time. 2-3 mounts before release
we should focus on testing, bugfixing, etc. - i.e. before RC1.

while keeping in mind that no system can be 100% ideal, and if we go too
far into stability we stifle new development, it is generally good to keep
the stable branch in a near-releasable state at all times in case we have to
do an emergency release due to some critical data-corruption or copyright bug.

just something to consider,
Hamish