[Geoserver-devel] Improving python (and other languages) scripting support for long running processes

GeoServer GeoServer Developer
1

Hi,
one of the things that the scripting language wps hooks are still not
very useful at is long running processes that might have to be run
in asynch mode, the ScriptProcess class receives a ProgressListener,
but it is not passed down to the script.

I would like to add this possibility, for starters, by extending the WPSHook.run
to get the ProgressListener, and leave the implementations free to make use of it.

In particular for the Python implementation I guess we could have the script
just get the ProgressListener object, and interact with it directly, no?
Alternatively, we could wrap it to make it more pythonic/simple (open to
suggestions here).

However… how do we expose that the execution method supports it?
Unlike Java, we cannot recognize the progress listener by the argument type.
I guess we could make it one of the inputs?

@process(
…
inputs={‘arg1’: (, ‘Arg1 description’),
‘arg2’: (, ‘Arg2 description’)}
‘arg3:’ (ProgressListener, 'The progress listener)}
)

Or else, we could have it be declared as a top level thing:

@process(
…
inputs = {…}
outputs = { … }
listener = ‘argName’
}

Ah, while I’m at it, another thing I want to check. What if the computation
is python finds some issue, we want to exit throwing a descriptive exception,
I guess all there is to it is to call inside the script:

raise Exception(‘My long winded explanation here’)

Right?

Cheers
Andrea

–

==
Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

2

I find this a good idea!

My suggestion would be to make use of **kwords arguments in python, so you can

Quote:

“You can use these along with named arguments too. The explicit arguments get values first and then everything else is passed to *args and *kwargs. The named arguments come first in the list. For example:

def table_things(titlestring, **kwargs)

“

So, the process session object always is the last argument, and you could also access it by kword:

http://freepythontips.wordpress.com/2013/08/04/args-and-kwargs-in-python-explained/

···

However… how do we expose that the execution method supports it?

Unlike Java, we cannot recognize the progress listener by the argument type.

I guess we could make it one of the inputs?

@process(

…

inputs={‘arg1’: (, ‘Arg1 description’),

‘arg2’: (, ‘Arg2 description’)}

‘arg3:’ (ProgressListener, 'The progress listener)}

)

Or else, we could have it be declared as a top level thing:

@process(

…

inputs = {…}

outputs = { … }

listener = ‘argName’

}

3

I thought in Java I checked if the implementation accepted a progress listener as the last input, and if so I would pass it in.
(That may of been an early draft that did not get off the workbench but it is still an idea).

Out of your two suggestions I like the “top level thing”, but would happily defer to Python coding conventions on the subject.
Jody

···

Jody Garnett

On Wed, May 21, 2014 at 8:38 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
one of the things that the scripting language wps hooks are still not
very useful at is long running processes that might have to be run
in asynch mode, the ScriptProcess class receives a ProgressListener,
but it is not passed down to the script.

I would like to add this possibility, for starters, by extending the WPSHook.run
to get the ProgressListener, and leave the implementations free to make use of it.

In particular for the Python implementation I guess we could have the script
just get the ProgressListener object, and interact with it directly, no?
Alternatively, we could wrap it to make it more pythonic/simple (open to
suggestions here).

However… how do we expose that the execution method supports it?
Unlike Java, we cannot recognize the progress listener by the argument type.
I guess we could make it one of the inputs?

@process(
…
inputs={‘arg1’: (, ‘Arg1 description’),
‘arg2’: (, ‘Arg2 description’)}
‘arg3:’ (ProgressListener, 'The progress listener)}
)

Or else, we could have it be declared as a top level thing:

@process(
…
inputs = {…}
outputs = { … }
listener = ‘argName’
}

Ah, while I’m at it, another thing I want to check. What if the computation
is python finds some issue, we want to exit throwing a descriptive exception,
I guess all there is to it is to call inside the script:

raise Exception(‘My long winded explanation here’)

Right?

Cheers
Andrea

–

==
Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

“Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free.”
http://p.sf.net/sfu/SauceLabs

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

4

On Wed, May 21, 2014 at 6:38 AM, Andrea Aime
<andrea.aime@anonymised.com>wrote:

Hi,
one of the things that the scripting language wps hooks are still not
very useful at is long running processes that might have to be run
in asynch mode, the ScriptProcess class receives a ProgressListener,
but it is not passed down to the script.

I would like to add this possibility, for starters, by extending the
WPSHook.run
to get the ProgressListener, and leave the implementations free to make
use of it.

In particular for the Python implementation I guess we could have the
script
just get the ProgressListener object, and interact with it directly, no?
Alternatively, we could wrap it to make it more pythonic/simple (open to
suggestions here).

However.. how do we expose that the execution method supports it?
Unlike Java, we cannot recognize the progress listener by the argument
type.
I guess we could make it one of the inputs?

@process(
  ...
  inputs={'arg1': (<arg1 type>, 'Arg1 description'),
          'arg2': (<arg2 type>, 'Arg2 description')}
          'arg3:' (ProgressListener, 'The progress listener)}
)

Or else, we could have it be declared as a top level thing:

@process(
...
  inputs = {...}
  outputs = { ... }
  listener = 'argName'
}

What about just setting a global variable in the script context named

"progress"? That way it becomes orthogonal to the process metadata and its
easy to get at for the script writer.

Ah, while I'm at it, another thing I want to check. What if the computation
is python finds some issue, we want to exit throwing a descriptive
exception,
I guess all there is to it is to call inside the script:

raise Exception('My long winded explanation here')

Right?

Yup, any exceptions thrown by the script will propagate back out. If the
script needs to throw a specific kind of exception (like WPSException) it
is the same process (after importing the java exception class).

Cheers
Andrea

--

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform
available
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
*Justin Deoliveira*
Vice President, Engineering | Boundless
jdeolive@anonymised.com
@j_deolive <https://twitter.com/j_deolive&gt;

5

On Wed, May 21, 2014 at 4:11 PM, Justin Deoliveira <
jdeolive@anonymised.com> wrote:

@process(

...
  inputs = {...}
  outputs = { ... }
  listener = 'argName'
}

What about just setting a global variable in the script context named

"progress"? That way it becomes orthogonal to the process metadata and its
easy to get at for the script writer.

Why not, but I'm wondering about multi-threading, would this global
variable be shared among running instances, or each one would have their
own private value?

Cheers
Andrea

--

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

6

On Wed, May 21, 2014 at 8:13 AM, Andrea Aime
<andrea.aime@anonymised.com>wrote:

On Wed, May 21, 2014 at 4:11 PM, Justin Deoliveira <
jdeolive@anonymised.com> wrote:

@process(

...
  inputs = {...}
  outputs = { ... }
  listener = 'argName'
}

What about just setting a global variable in the script context named

"progress"? That way it becomes orthogonal to the process metadata and its
easy to get at for the script writer.

Why not, but I'm wondering about multi-threading, would this global
variable be shared among running instances, or each one would have their
own private value?

You're right, that would not work in a multi threaded case, the only way i
can think of is pass it in as a function argument or use a thread local. We
could simply support an argument named "progress" without explicitly
declaring it in the process metadata? Since arguments are matched to the
metadata by key I believe it would work.

Cheers
Andrea

--

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

--
*Justin Deoliveira*
Vice President, Engineering | Boundless
jdeolive@anonymised.com
@j_deolive <https://twitter.com/j_deolive&gt;

7

On Wed, May 21, 2014 at 4:27 PM, Justin Deoliveira <
jdeolive@anonymised.com> wrote:

You're right, that would not work in a multi threaded case, the only way i
can think of is pass it in as a function argument or use a thread local. We
could simply support an argument named "progress" without explicitly
declaring it in the process metadata? Since arguments are matched to the
metadata by key I believe it would work.

Works for me. So we check if there is a progress argument, that is not
already declared as a true input of the
process (backwards compatibility, you never know) and pass it a
ProgressListener.

How about the usage from Python, is ProgressListener edible enough, or do
we have to wrap it?
https://github.com/geotools/geotools/blob/630c0aff44c4982f4202bef74205df081d999d18/modules/library/opengis/src/main/java/org/opengis/util/ProgressListener.java

Cheers
Andrea

--

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

8

On Wed, May 21, 2014 at 8:32 AM, Andrea Aime
<andrea.aime@anonymised.com>wrote:

On Wed, May 21, 2014 at 4:27 PM, Justin Deoliveira <
jdeolive@anonymised.com> wrote:

You're right, that would not work in a multi threaded case, the only way
i can think of is pass it in as a function argument or use a thread local.
We could simply support an argument named "progress" without explicitly
declaring it in the process metadata? Since arguments are matched to the
metadata by key I believe it would work.

Works for me. So we check if there is a progress argument, that is not
already declared as a true input of the
process (backwards compatibility, you never know) and pass it a
ProgressListener.

How about the usage from Python, is ProgressListener edible enough, or do
we have to wrap it?

https://github.com/geotools/geotools/blob/630c0aff44c4982f4202bef74205df081d999d18/modules/library/opengis/src/main/java/org/opengis/util/ProgressListener.java

Good question. The only thing that worries me about the interface is the

use of InternationalString which isn't convenient to use. There is
setDescription but that is deprecated. It should be easy enough to just
provide an implementation of ProgressListener in the script module that
exposes task as a string (and perhaps any convenience methods we want) and
then just pass that in.

Cheers
Andrea

--

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

--
*Justin Deoliveira*
Vice President, Engineering | Boundless
jdeolive@anonymised.com
@j_deolive <https://twitter.com/j_deolive&gt;

9

On Wed, May 21, 2014 at 4:38 PM, Justin Deoliveira <
jdeolive@anonymised.com> wrote:

On Wed, May 21, 2014 at 8:32 AM, Andrea Aime <andrea.aime@anonymised.com
> wrote:

On Wed, May 21, 2014 at 4:27 PM, Justin Deoliveira <
jdeolive@anonymised.com> wrote:

You're right, that would not work in a multi threaded case, the only way
i can think of is pass it in as a function argument or use a thread local.
We could simply support an argument named "progress" without explicitly
declaring it in the process metadata? Since arguments are matched to the
metadata by key I believe it would work.

Works for me. So we check if there is a progress argument, that is not
already declared as a true input of the
process (backwards compatibility, you never know) and pass it a
ProgressListener.

How about the usage from Python, is ProgressListener edible enough, or do
we have to wrap it?

https://github.com/geotools/geotools/blob/630c0aff44c4982f4202bef74205df081d999d18/modules/library/opengis/src/main/java/org/opengis/util/ProgressListener.java

Good question. The only thing that worries me about the interface is the

use of InternationalString which isn't convenient to use. There is
setDescription but that is deprecated. It should be easy enough to just
provide an implementation of ProgressListener in the script module that
exposes task as a string (and perhaps any convenience methods we want) and
then just pass that in.

Cool, will do that

Cheers
Andrea

--

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

10

I did try and make a Text implementation of InternationalString that is easier to use.

···

Jody Garnett

On Wed, May 21, 2014 at 12:09 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

“Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos.
Get unparalleled scalability from the best Selenium testing platform available
Simple to use. Nothing to install. Get started now for free.”
http://p.sf.net/sfu/SauceLabs

Geoserver-devel mailing list
Geoserver-devel@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

On Wed, May 21, 2014 at 4:38 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

Cool, will do that

Cheers

Andrea

–

==
Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

On Wed, May 21, 2014 at 8:32 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Good question. The only thing that worries me about the interface is the use of InternationalString which isn’t convenient to use. There is setDescription but that is deprecated. It should be easy enough to just provide an implementation of ProgressListener in the script module that exposes task as a string (and perhaps any convenience methods we want) and then just pass that in.

On Wed, May 21, 2014 at 4:27 PM, Justin Deoliveira <jdeolive@anonymised.com> wrote:

Works for me. So we check if there is a progress argument, that is not already declared as a true input of the
process (backwards compatibility, you never know) and pass it a ProgressListener.

How about the usage from Python, is ProgressListener edible enough, or do we have to wrap it?
https://github.com/geotools/geotools/blob/630c0aff44c4982f4202bef74205df081d999d18/modules/library/opengis/src/main/java/org/opengis/util/ProgressListener.java

You’re right, that would not work in a multi threaded case, the only way i can think of is pass it in as a function argument or use a thread local. We could simply support an argument named “progress” without explicitly declaring it in the process metadata? Since arguments are matched to the metadata by key I believe it would work.

11

On Wed, May 21, 2014 at 6:14 PM, Jody Garnett <jody.garnett@anonymised.com>wrote:

I did try and make a Text implementation of InternationalString that is
easier to use.

It seems to me the only truly python friendly method in that utility class
is:

  public static InternationalString text(String english){
        return new SimpleInternationalString( english );
  }

It would still look "ugly" in python I guess, compared to just calling
method of a bean, which
are turned into python object properties:
http://www.jython.org/archive/21/docs/properties.html

Cheers
Andrea

--

Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------