[GRASS-dev] Python and delayed or missing stderr output

GRASS GRASS developer
1

Hi,

AFAIK Python buffers (i.e. effectively delays) stderr output which is
unhelpful in the GRASS GIS context.

Searching for a solution, I found this reference: sys.stdout.flush()
e.g.
https://stackoverflow.com/questions/10019456/usage-of-sys-stdout-flush-method

Would it be possible to add that to the GRASS GIS Python API?

thanks,
markusN

2

* Markus Neteler <neteler@osgeo.org> [2018-09-05 10:43:23 +0200]:

Hi,

AFAIK Python buffers (i.e. effectively delays) stderr output which is
unhelpful in the GRASS GIS context.

Searching for a solution, I found this reference: sys.stdout.flush()
e.g.
https://stackoverflow.com/questions/10019456/usage-of-sys-stdout-flush-method

Would it be possible to add that to the GRASS GIS Python API?

And would this then make Python scripts, by default, to issue any print
statements (for debugging purposes or else) immediately?

I am unsure, but I think I see exactly this "problem" sometimes, when
scripting. The order of what is requested to print out is sometimes mixed.

Thanks Markus.

3

On Wed, Sep 5, 2018 at 10:43 AM Markus Neteler <neteler@osgeo.org> wrote:

Hi,

AFAIK Python buffers (i.e. effectively delays) stderr output which is
unhelpful in the GRASS GIS context.

Searching for a solution, I found this reference: sys.stdout.flush()
e.g.

https://stackoverflow.com/questions/10019456/usage-of-sys-stdout-flush-method

That should be
sys.stderr.flush()
right?

Regarding python’s multiprocessing, it makes sense that stderr is suppressed because you would not know which process process produced which stderr (or stdout) message. There must be a method in python to redirect stderr and stdout for each process to a unique file, otherwise it would not be possible to find out why some process failed.

Such a method would need to be implemented in each module usings python’s multiprocessing.

Markus M

4

On Sun, Sep 9, 2018 at 9:47 PM Markus Metz <markus.metz.giswork@gmail.com> wrote:

On Wed, Sep 5, 2018 at 10:43 AM Markus Neteler <neteler@osgeo.org> wrote:

Hi,

AFAIK Python buffers (i.e. effectively delays) stderr output which is
unhelpful in the GRASS GIS context.

Searching for a solution, I found this reference: sys.stdout.flush()
e.g.
https://stackoverflow.com/questions/10019456/usage-of-sys-stdout-flush-method

That should be
sys.stderr.flush()
right?

Regarding python’s multiprocessing, it makes sense that stderr is suppressed because you would not know which process process produced which stderr (or stdout) message. There must be a method in python to redirect stderr and stdout for each process to a unique file, otherwise it would not be possible to find out why some process failed.

Such a method would need to be implemented in each module usings python’s multiprocessing.

maybe worth a try also with stderr:

https://stackoverflow.com/questions/1501651/log-output-of-multiprocessing-process

i.e. within the function that is the target of multiprocessing.Process

sys.stderr = open(str(os.getpid()) + “.err.log”, “w”)

and it seems to be a good idea to always sys.stderr.flush() at the end of the subprocess

Markus M