[GRASS-dev] Add support for Python3

Pietro2 · July 24, 2015, 10:42am

Dear devs,

Next week I would like to start the merge between my local changes and
trunk to support Python3.
As reported in the ticket: https://trac.osgeo.org/grass/ticket/2708

I've tested the changes with python2.6, python2.7 and python3.4.
So far I've not found regressions in python2.X.

The status quo for python3 is:
- the binding of the C GRASS functions is not working;
- several python modules (~90) are not readable/usable, I've already
started to port them (~20)

My idea for the porting is to gradually introduce changes per steps:
- Monday 27, change only the grass-init and the gunittest;
- If not new problems connected to previous changes raise, Wendesday
29 I commit the grass/script stuff
- If not new problems connected to previous changes raise, I will
start updating the GRASS modules.

In the meanwhile I try to get the ctypes working on PY3.

Are you ok with this schedule?
Other better options?

Let me know

Pietro

Yann4 · July 24, 2015, 12:42pm

+1

On Fri, Jul 24, 2015 at 4:12 PM Pietro <peter.zamb@gmail.com> wrote:

Dear devs,

Next week I would like to start the merge between my local changes and
trunk to support Python3.
As reported in the ticket: https://trac.osgeo.org/grass/ticket/2708

I’ve tested the changes with python2.6, python2.7 and python3.4.
So far I’ve not found regressions in python2.X.

The status quo for python3 is:

the binding of the C GRASS functions is not working;

several python modules (~90) are not readable/usable, I’ve already
started to port them (~20)

My idea for the porting is to gradually introduce changes per steps:

Monday 27, change only the grass-init and the gunittest;

If not new problems connected to previous changes raise, Wendesday
29 I commit the grass/script stuff

If not new problems connected to previous changes raise, I will
start updating the GRASS modules.

In the meanwhile I try to get the ctypes working on PY3.

Are you ok with this schedule?
Other better options?

Let me know

Pietro

grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

lucadelu · July 27, 2015, 6:47am

On 24 July 2015 at 12:42, Pietro <peter.zamb@gmail.com> wrote:

Dear devs,

Ciao Pietro,

Next week I would like to start the merge between my local changes and
trunk to support Python3.
As reported in the ticket: https://trac.osgeo.org/grass/ticket/2708

I've tested the changes with python2.6, python2.7 and python3.4.
So far I've not found regressions in python2.X.

The status quo for python3 is:
- the binding of the C GRASS functions is not working;
- several python modules (~90) are not readable/usable, I've already
started to port them (~20)

My idea for the porting is to gradually introduce changes per steps:
- Monday 27, change only the grass-init and the gunittest;
- If not new problems connected to previous changes raise, Wendesday
29 I commit the grass/script stuff
- If not new problems connected to previous changes raise, I will
start updating the GRASS modules.

In the meanwhile I try to get the ctypes working on PY3.

Are you ok with this schedule?

yes, let us know when you commit the changes to test them

Other better options?

Let me know

Pietro

--
ciao
Luca

http://gis.cri.fmach.it/delucchi/
www.lucadelu.org

Pietro2 · July 27, 2015, 11:59am

Hi Luca,

On Mon, Jul 27, 2015 at 8:47 AM, Luca Delucchi <lucadeluge@gmail.com> wrote:

On 24 July 2015 at 12:42, Pietro <peter.zamb@gmail.com> wrote:

Are you ok with this schedule?

yes, let us know when you commit the changes to test them

ok, I've commited the changes in lib/init/grass.py and in the
lib/python/gunittest/*

Let me know if you have troubles.

Pietro

neteler · July 27, 2015, 2:24pm

On Mon, Jul 27, 2015 at 1:59 PM, Pietro <peter.zamb@gmail.com> wrote:
...

ok, I've commited the changes in lib/init/grass.py and in the
lib/python/gunittest/*

Please post to the list how to switch to Python3 in case that multiple
versions are available.
Like this more testers may join.

thanks
Markus

Pietro2 · July 27, 2015, 7:58pm

Hi Markus,

On Mon, Jul 27, 2015 at 4:24 PM, Markus Neteler <neteler@osgeo.org> wrote:

On Mon, Jul 27, 2015 at 1:59 PM, Pietro <peter.zamb@gmail.com> wrote:
...

ok, I've commited the changes in lib/init/grass.py and in the
lib/python/gunittest/*

Please post to the list how to switch to Python3 in case that multiple
versions are available.
Like this more testers may join.

You are righ, here some instruction to test and find bugs...

You must have python3 installed on your machine, from a terminal

{{{
python3 /usr/bin/grass71 -getxt ~/docdat/gis/nc_spm_08_grass7/py34 -c
}}}

then start a python interpreter and execute some code
[Note1: this example should not work on your machine since I didn't
commit this part so far;
Note2: GRASS with python3 it is working only in text mode no GUI!]:

{{{
$ python3

from grass.script import core as gcore
gcore.gisenv()

{'GISDBASE': '/home/pietro/docdat/gis', 'GUI': 'gtext', 'MAPSET':
'py34', 'LOCATION_NAME': 'nc_spm_08_grass7'}
}}}

To check if the result in python2 is the same then open a new terminal
and execute

{{{
$ python2 /usr/bin/grass71 -getxt ~/docdat/gis/nc_spm_08_grass7/py27 -c
}}}

{{{
$ python2

from grass.script import core as gcore
gcore.gisenv()

{u'LOCATION_NAME': unc_spm_08_grass7', u'GISDBASE':
u'/home/pietro/docdat/gis', u'MAPSET': u'py27', u'GUI': u'gtext'}
}}}

You can also start executing Vaclav's unittest library, go to a directory:

{{{
$ cd lib/python/script
$ python3 -m grass.gunittest.main --location nc_spm_08_grass7 --location-type nc
test_start_command_functions_nc from . failed
test_start_command_functions from . failed
test_doctests from . failed

Executed 3 test files in 0:00:09.962775.

From them 0 files (0%) were successful and 3 files (100%) failed.

$ python2 -m grass.gunittest.main --location nc_spm_08_grass7 --location-type nc

Executed 3 test files in 0:00:10.358305.

From them 3 files (100%) were successful and 0 files (0%) failed.

}}}

or execute a specific test file with:

{{{
$ python3 testsuite/test_start_command_functions_nc.py
}}}

So far the test in the script directory are still failing I'm working
to fix them...

All the best

Pietro

Glynn_Clements1 · July 27, 2015, 8:47pm

Markus Neteler wrote:

> ok, I've commited the changes in lib/init/grass.py and in the
> lib/python/gunittest/*

Please post to the list how to switch to Python3 in case that multiple
versions are available.
Like this more testers may join.

It depends upon how the different Python versions are installed.

GRASS' scripts use the shebang:

#!/usr/bin/env python

so the first program named "python" in $PATH will be used.

If Python 3.x is installed as e.g. /usr/bin/python3, it usually
suffices to create a directory containing a symlink from python to
/usr/bin/python3 and put that directory at the start of $PATH. E.g.

  mkdir python3
  ln -s /usr/bin/python3 python3/python
  export PATH=`pwd`/python3:$PATH

--
Glynn Clements <glynn@gclements.plus.com>

Pietro2 · July 29, 2015, 2:31pm

Hi all,

On Mon, Jul 27, 2015 at 8:47 AM, Luca Delucchi <lucadeluge@gmail.com> wrote:

On 24 July 2015 at 12:42, Pietro <peter.zamb@gmail.com> wrote:

Are you ok with this schedule?

yes, let us know when you commit the changes to test them

I tested using the current testsuite, please tell me if you find bugs
or better write a test that highlights the problem and I will try to
fix it as soon as I can.

All the best

Pietro

Glynn_Clements1 · July 30, 2015, 2:34pm

Pietro wrote:

ok, I've now synchronized my local changes in script/core to trunk:

In parser() in lib/python/script/core.py:

s = p.communicate()[0].decode()

the .decode() shouldn't be there; the output should be kept as bytes.
This may require changing _parse_opts().

Likewise for parse_key_val() in utils.py.

More generally, nothing in grass.script should be using unicode
values. Arguments which happen to be unicode values can be encode()d
to bytes, but arguments which are bytes must be used as-is (no
bytes-unicode-bytes round trips) and data obtained from files, pipes,
environment variables, OS functions, etc should be returned as bytes.

--
Glynn Clements <glynn@gclements.plus.com>

Pietro2 · August 4, 2015, 6:14am

Hi Glynn,

On Thu, Jul 30, 2015 at 4:34 PM, Glynn Clements wrote:

Pietro wrote:

ok, I've now synchronized my local changes in script/core to trunk:

In parser() in lib/python/script/core.py:

s = p.communicate()[0].decode()

the .decode() shouldn't be there; the output should be kept as bytes.
This may require changing _parse_opts().

Likewise for parse_key_val() in utils.py.

More generally, nothing in grass.script should be using unicode
values. Arguments which happen to be unicode values can be encode()d
to bytes, but arguments which are bytes must be used as-is (no
bytes-unicode-bytes round trips) and data obtained from files, pipes,
environment variables, OS functions, etc should be returned as bytes.

ok, I try to avoid to use unicode in script with the following changes:

https://trac.osgeo.org/grass/changeset/65804

I've stopped to go on with the syncronization of my local changes and
trunk since I noticed that from the 27th of July the tests are not
running anymore:

http://fatra.cnr.ncsu.edu/grassgistests/summary_report/nc/index.html

I don't understand why, but I have the suspect that the recent changes
done in lib/init/grass.py and gunittest:

https://trac.osgeo.org/grass/changeset/65780
https://trac.osgeo.org/grass/changeset/65782

have broken something.

I will not go further on the development of python2/python3 until this
issue will be fix.

All the best.

Pietro

wenzeslaus · August 5, 2015, 2:30pm

On Tue, Aug 4, 2015 at 2:14 AM, Pietro <peter.zamb@gmail.com> wrote:

I’ve stopped to go on with the syncronization of my local changes and
trunk since I noticed that from the 27th of July the tests are not
running anymore:

http://fatra.cnr.ncsu.edu/grassgistests/summary_report/nc/index.html

I don’t understand why, but I have the suspect that the recent changes
done in lib/init/grass.py and gunittest:

https://trac.osgeo.org/grass/changeset/65780
https://trac.osgeo.org/grass/changeset/65782

have broken something.

I will not go further on the development of python2/python3 until this
issue will be fix.

The issue was caused by missing g.region call in one of the tests [1, 2].

However, now I see at least 3 tests which were not failing before [3, 4, 5]. It seems like a trivial issue caused by some automatic changes (percent operator is missing). This issue should be visible on any machine.

Also the error messages seems to be inappropriate. This should just fail and not report an error (perhaps an except statement without an exception specified?). And also the formatting is wrong since <> is missing around the vector map name. Perhaps all three messages should be the same.

To run just one selected test use, for example:

cd scripts/v.what.strds/testsuite
python test_what_strds.py TestWhatStrds.test_output

Vaclav

[1] https://lists.osgeo.org/pipermail/grass-dev/2015-May/075025.html
[2] https://lists.osgeo.org/pipermail/grass-dev/2015-August/075900.html
[3] http://fatra.cnr.ncsu.edu/grassgistests/reports_for_date-2015-08-04-16-20/report_for_nc_basic_spm_grass7_nc/temporal/t.vect.observe.strds/test.t.vect.observe.strds.relative/index.html
[4] http://fatra.cnr.ncsu.edu/grassgistests/reports_for_date-2015-08-04-16-20/report_for_nc_basic_spm_grass7_nc/temporal/t.vect.observe.strds/test.t.vect.observe.strds/index.html
[5] http://fatra.cnr.ncsu.edu/grassgistests/reports_for_date-2015-08-04-16-20/report_for_nc_basic_spm_grass7_nc/scripts/v.what.strds/test_what_strds/index.html

neteler · August 5, 2015, 2:49pm

On Wed, Aug 5, 2015 at 4:30 PM, Vaclav Petras <wenzeslaus@gmail.com> wrote:
...

To run just one selected test use, for example:

cd scripts/v.what.strds/testsuite
python test_what_strds.py TestWhatStrds.test_output

Yes,

GRASS 7.1.svn (nc_spm_08_grass7):~/software/grass71/scripts/v.what.strds/testsuite

python test_what_strds.py TestWhatStrds.test_output

...
Traceback (most recent call last):
  File "test_what_strds.py", line 50, in test_output
    output="what_strds", overwrite=True)
  File "/home/neteler/software/grass71/dist.x86_64-unknown-linux-gnu/etc/python/grass/gunittest/case.py",
line 1139, in assertModule
    self.fail(self._formatMessage(msg, stdmsg))
AssertionError: Running <v.what.strds> module ended with non-zero
return code (1)
Called: v.what_strds(input='points', strds=['A'], output='what_strds',
overwrite=True)
See the folowing errors:
DB settings already defined, nothing to do
Traceback (most recent call last):
  File "/home/neteler/software/grass71/dist.x86_64-unknown-linux-gnu/scripts/v.db.addtable",
line 150, in <module>
    sys.exit(main())
  File "/home/neteler/software/grass71/dist.x86_64-unknown-linux-gnu/scripts/v.db.addtable",
line 129, in main
    table = '%s.%s' (schema, table)
TypeError: 'str' object is not callable
ERROR: Impossible add table to vector what_strds

this part looks suspicious:
... strds=['A'] ...

the [..] might cause the issue?

Markus

wenzeslaus · August 5, 2015, 4:07pm

On Wed, Aug 5, 2015 at 10:49 AM, Markus Neteler <neteler@osgeo.org> wrote:

table = '%s.%s' (schema, table)
TypeError: 'str' object is not callable

As I said, I think it is missing percent operator. Probably trivial to fix
but we should understand how it happen to be in the source code.

String formatting:

'%s' % ('abc')

Function call:

'%s' ('abc')

Glynn_Clements1 · August 6, 2015, 1:09pm

Vaclav Petras wrote:

> table = '%s.%s' (schema, table)
> TypeError: 'str' object is not callable
>

As I said, I think it is missing percent operator. Probably trivial to fix
but we should understand how it happen to be in the source code.

It's a bug which was introduced in r56701:

https://trac.osgeo.org/grass/changeset/56701/grass/trunk/scripts/v.db.addtable/v.db.addtable.py

It's unrelated to the Python3 conversion.

--
Glynn Clements <glynn@gclements.plus.com>

Glynn_Clements1 · August 6, 2015, 1:16pm

Vaclav Petras wrote:

However, now I see at least 3 tests which were not failing before [3, 4,
5]. It seems like a trivial issue caused by some automatic changes (percent
operator is missing). This issue should be visible on any machine.

The bug which is causing the test to fail has been present in
v.db.addtable for 2 years (since r56701).

If the tests only started failing recently, it can only be because of
changes to how v.db.addtable was invoked.

A database connection having a "schema" option where none existed
before would seem likely. The error arises from:

if schema is not '':
table = '%s.%s' (schema, table)

where "schema" is set by:

    kv = grass.db_connection()
    database = kv['database']
    driver = kv['driver']
    schema = kv['schema']

--
Glynn Clements <glynn@gclements.plus.com>

annakrat · August 26, 2015, 2:06am

On Thu, Aug 6, 2015 at 9:16 AM, Glynn Clements <glynn@gclements.plus.com>
wrote:

Vaclav Petras wrote:

> However, now I see at least 3 tests which were not failing before [3, 4,
> 5]. It seems like a trivial issue caused by some automatic changes
(percent
> operator is missing). This issue should be visible on any machine.

The bug which is causing the test to fail has been present in
v.db.addtable for 2 years (since r56701).

If the tests only started failing recently, it can only be because of
changes to how v.db.addtable was invoked.

A database connection having a "schema" option where none existed
before would seem likely. The error arises from:

if schema is not '':
table = '%s.%s' (schema, table)

The problem is related to the Python 3 support - the condition is wrong and
schema is now empty unicode string instead of empty string (coming from
db_connection()). I am not sure if unicode here is what we want, but that's
a different problem, the condition works for any empty string now.

Anna

where "schema" is set by:

    kv = grass.db_connection()
    database = kv['database']
    driver = kv['driver']
    schema = kv['schema']

--
Glynn Clements <glynn@gclements.plus.com>
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev