[GRASS-dev] Continuos output from grass python script of subprocess

GRASS GIS GRASS GIS developer
1

Hi grass-devs,
could you tell me why this simple example script doesn’t show me the output of my program that I’m launching through subprocess? It’s only giving me the output once the program has finished. As the program runs for quite some time, I would like to see the output while the script is running through the grass module interface. (In the actual script, I’m preparing some input and postprocess output before/after the program has run.)

Any hints would be highly appreciated.
Thanks and best regards,
Michel

#!/usr/bin/env python

#%module
#% description: Testing
#% keywords: raster, statistics
#%end

#%option
#% key: myprogram
#% type: string
#% key_desc: name
#% description: Any long running program with continous output to stdout
#%end

import sys
import subprocess

import grass.script as grass

def main():
    # put code here
    p = subprocess.Popen(options['myprogram'], shell=True,
                          stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    while True:
        nextline = p.stdout.readline()
        if nextline == '' and p.poll() != None:
            break
        grass.message(nextline)
    sys.stdout.flush()
    return 0

if __name__ == "__main__":
        
    # normal start
    options, flags = grass.parser()
    sys.exit(main())

2

Michel Wortmann wrote:

could you tell me why this simple example script doesn�t show me the
output of my program that I�m launching through subprocess? It�s
only giving me the output once the program has finished. As the
program runs for quite some time, I would like to see the output
while the script is running through the grass module interface. (In
the actual script, I�m preparing some input and postprocess output
before/after the program has run.)

The most likely reason is that the child program's stdout is
fully-buffered, meaning that the stdio functions will buffer the data
and only pass it to the OS when the buffer is full (for GNU libc, the
default buffer size is 8192 bytes).

Typically, stdout is line-buffered if it is associated with a
terminal, and fully buffered otherwise (e.g. if it is associated with
a file or pipe).

The exact wording of the C standard is (§7.19.3p7)

  As initially opened, the standard error stream is not fully
  buffered; the standard input and standard output streams are
  fully buffered if and only if the stream can be determined not
  to refer to an interactive device.

If the program is one you can modify, you can force line buffering
using the setvbuf() function, or you can explicitly flush the buffer
after each line with the fflush() function. Or you could write
progress output to stderr, which is typically unbuffered by default.

If the script doesn't need to process the program's output you could
just remove the stdout=subprocess.PIPE, allowing the child process to
inherit the script's stdout.

If the script needs to process the program's output, you can't modify
the program, and you're using Linux, you could use Python's "pty"
module (which is only available on Unix and only supported on Linux)
to create a pseudo-tty (pty), and set the child process' stdout to the
pty slave. This will cause the stdio library to make stdout
line-buffered.

--
Glynn Clements <glynn@gclements.plus.com>

3

Dear Glynn,
thank you ever so much for this detailed answer. Yes the program's buffering was the problem, I'm running a fortran executable and the flush() command did the trick.
Best regards,
Michel

On 08/11/2014 10:52 AM, Glynn Clements wrote:

Michel Wortmann wrote:

could you tell me why this simple example script doesn�t show me the
output of my program that I�m launching through subprocess? It�s
only giving me the output once the program has finished. As the
program runs for quite some time, I would like to see the output
while the script is running through the grass module interface. (In
the actual script, I�m preparing some input and postprocess output
before/after the program has run.)

The most likely reason is that the child program's stdout is
fully-buffered, meaning that the stdio functions will buffer the data
and only pass it to the OS when the buffer is full (for GNU libc, the
default buffer size is 8192 bytes).

Typically, stdout is line-buffered if it is associated with a
terminal, and fully buffered otherwise (e.g. if it is associated with
a file or pipe).

The exact wording of the C standard is (§7.19.3p7)

  As initially opened, the standard error stream is not fully
  buffered; the standard input and standard output streams are
  fully buffered if and only if the stream can be determined not
  to refer to an interactive device.

If the program is one you can modify, you can force line buffering
using the setvbuf() function, or you can explicitly flush the buffer
after each line with the fflush() function. Or you could write
progress output to stderr, which is typically unbuffered by default.

If the script doesn't need to process the program's output you could
just remove the stdout=subprocess.PIPE, allowing the child process to
inherit the script's stdout.

If the script needs to process the program's output, you can't modify
the program, and you're using Linux, you could use Python's "pty"
module (which is only available on Unix and only supported on Linux)
to create a pseudo-tty (pty), and set the child process' stdout to the
pty slave. This will cause the stdio library to make stdout
line-buffered.