[GRASS5] Libgrass_gis on Mingw - open.c patch

GRASS GIS GRASS GIS developer
1

Hi all!
This patch solves the problem of binary files read/write under win32.
Practically forces the c runtime to binary-mode file translation as
default.
It is not the cleanest way but it is simple.
More info at: http://www.mingw.org/MinGWiki/index.php/binary

Cheers,

Javier Yebrin

Javier A. Yebrin wrote:

Finally I've got most of grass modules compiled under Mingw/Msys. Then
started to test them (first the raster modules, of course), and found
some strange behaviour depending on the studied raster. I've used the
Spearfish database and ran r.sum for all the present rasters (using
spot.image to define the current region). These are the results(first
line the raster name, then the stdout or
stderr):

aspect
WARNING: Fail of initial read of compressed file [aspect in PERMANENT]

You may see that the response changes from one type of raster to
another, whether it is compresed or not. I'm trying to debug this
module at the moment...

In the meanwhile, do you have any ideas?

The usual reason for that error message on Windows is that something is
performing CRLF->LF conversion, which is inappropriate for a binary file.

ISTR that the MinGW runtime includes a function to enable/disable the
conversions. You need to ensure that all I/O is done without conversions.

--
Glynn Clements <glynn@gclements.plus.com>

(attachments)

diff_libgrass_gis_2.txt (541 Bytes)

2

Javier A. Yebrin wrote:

Hi all!
This patch solves the problem of binary files read/write under win32.
Practically forces the c runtime to binary-mode file translation as default.
It is not the cleanest way but it is simple.
More info at: http://www.mingw.org/MinGWiki/index.php/binary

Cheers,

Javier Yebrin

Isn't it better to use 'b' as mode for fdopen?

Do I understand it correctly that text files on windows will be also written with LF only so that a mapset can be accessed from UNIX without problems?

Could it happen that also files which should be opened as text will be opened as binary? For example text input for modules.

Radim

Javier A. Yebrin wrote:

Finally I've got most of grass modules compiled under Mingw/Msys. Then started to test them (first the raster modules, of course), and found some strange behaviour depending on the studied raster. I've used the Spearfish database and ran r.sum for all the present rasters (using spot.image to define the current region). These are the results(first line the raster name, then the stdout or
stderr):

aspect
WARNING: Fail of initial read of compressed file [aspect in PERMANENT]

You may see that the response changes from one type of raster to another, whether it is compresed or not. I'm trying to debug this module at the moment...

In the meanwhile, do you have any ideas?

The usual reason for that error message on Windows is that something is
performing CRLF->LF conversion, which is inappropriate for a binary file.

ISTR that the MinGW runtime includes a function to enable/disable the
conversions. You need to ensure that all I/O is done without conversions.

------------------------------------------------------------------------

diff -u -r /home/javier/grass-6.1.cvs_src_snapshot_2005_05_07/lib/gis/open.c /home/javier/static_grass/lib/gis/open.c
--- /home/javier/grass-6.1.cvs_src_snapshot_2005_05_07/lib/gis/open.c Tue Nov 9 12:20:13 2004
+++ /home/javier/static_grass/lib/gis/open.c Thu May 12 15:14:09 2005
@@ -83,6 +83,13 @@
#include <unistd.h>
#include <fcntl.h>
+#ifdef __MINGW32__ +#include <stdlib.h> /* _fmode */
+#include <fcntl.h> /* _O_BINARY */
+#undef _fmode
+int _fmode = _O_BINARY;
+#endif +
int G__open (
     char *element,
     char *name,

3

Radim Blazek wrote:

> This patch solves the problem of binary files read/write under win32.
> Practically forces the c runtime to binary-mode file translation as
> default.
> It is not the cleanest way but it is simple.
> More info at: http://www.mingw.org/MinGWiki/index.php/binary

Isn't it better to use 'b' as mode for fdopen?

By default MinGW will perform EOL translation for files which are
accessed via the POSIX compatibility functions (open, read, write),
not just for the ANSI stdio functions.

Do I understand it correctly that text files on windows will be also
written with LF only so that a mapset can be accessed from UNIX without
problems?

Could it happen that also files which should be opened as text will be
opened as binary? For example text input for modules.

All files should be opened as binary on all platforms. GRASS should do
its own EOL translation where necessary, so that files created on
Windows will work on Unix.

--
Glynn Clements <glynn@gclements.plus.com>

4

Javier A. Yebrin wrote:

Hi all!
This patch solves the problem of binary files read/write under win32.
Practically forces the c runtime to binary-mode file translation as default.
It is not the cleanest way but it is simple.
More info at: http://www.mingw.org/MinGWiki/index.php/binary

Cheers,

Javier Yebrin

I have submitted the patch to CVS HEAD.

Can you summarize what is currently working on Windows/MinGW?

Thanks

Radim

5

These are the raster modules currently built on MINGW (note that built
does'nt mean working, nor fully tested):

r.average
r.basins.fill
r.bilinear
r.buffer
r.cats
r.circle
r.clump
r.coin
r.colors
r.composite
r.compress
r.contour
r.cost
r.covar
r.cross
r.describe
r.distance
r.drain
r.fill.dir
r.flow
r.grow
r.his
r.in.arc
r.in.ascii
r.in.bin
r.in.gdal
r.in.gridatb
r.in.mat
r.in.poly
r.info
r.kappa
r.le.patch
r.los
r.mapcalc_grass
r.median
r.mfilter
r.mode
r.neighbors
r.null
r.out.arc
r.out.ascii
r.out.bin
r.out.gridatb
r.out.mat
r.out.mpeg
r.out.pov
r.out.ppm3
r.param.scale
r.patch
r.proj
r.quant
r.random.cells
r.random.surface
r.reclass
r.recode
r.report
r.resample
r.rescale
r.rescale.eq
r.series
r.slope.aspect
r.statistics
r.stats
r.sum
r.sun
r.sunmask
r.surf.area
r.surf.contour
r.surf.idw
r.texture
r.thin
r.topidx
r.topmodel
r.transect
r.univar
r.water.outlet
r.watershed
r.what

-----Original Message-----
From: Radim Blazek [mailto:blazek@itc.it]
Sent: Monday, May 16, 2005 9:52 AM
To: Javier A. Yebrin
Cc: grass5@grass.itc.it
Subject: Re: [GRASS5] Libgrass_gis on Mingw - open.c patch

Javier A. Yebrin wrote:
> Hi all!
> This patch solves the problem of binary files read/write
under win32.
> Practically forces the c runtime to binary-mode file translation as
> default. It is not the cleanest way but it is simple.
> More info at: http://www.mingw.org/MinGWiki/index.php/binary
>
> Cheers,
>
> Javier Yebrin

I have submitted the patch to CVS HEAD.

Can you summarize what is currently working on Windows/MinGW?

Thanks

Radim

6

These are the raster modules currently built on MINGW (note that built
does'nt mean working, nor fully tested):

how about a list of what doesn't build?

thanks,
Hamish

7

how about a list of what doesn't build?

Something like:

r.digit
r.out.png
r.out.ppm3
r.out.tiff
r.random
r.region
r.resamp.rst
r.surf.fractal
r.surf.gauss
r.surf.idw2
r.surf.random
r.terraflow
r.timestamp
r.to.vect

Some of them will not work because they need a GUI (r.digit). Others maybe
because of wrong configure flags (r.out.***).
Cheers,

Javier

8

Javier A. Yebrin wrote:

> how about a list of what doesn't build?
>
Something like:

r.digit
r.out.png
r.out.ppm3
r.out.tiff
r.random
r.region
r.resamp.rst
r.surf.fractal
r.surf.gauss
r.surf.idw2
r.surf.random
r.terraflow
r.timestamp
r.to.vect

Some of them will not work because they need a GUI (r.digit). Others maybe
because of wrong configure flags (r.out.***).

Can you provide the actual build errors for the above modules?

--
Glynn Clements <glynn@gclements.plus.com>

9

Hi,
I'll send the build errors as soon I get my work organized. I've started
with some real testing and recompilation. The current module undergoing this
test is r.average, which is presenting some problems because of system calls
to r.stats and r.recode.
I'm thinking of writing the r.stats/recode modules as functions inside
r.average. Do you think this is feasable?

Javier

-----Original Message-----
From: Glynn Clements [mailto:glynn@gclements.plus.com]
Sent: Tuesday, May 17, 2005 6:01 PM
To: Javier A. Yebrin
Cc: 'Hamish'; grass5@grass.itc.it
Subject: RE: [GRASS5] Libgrass_gis on Mingw - open.c patch

Javier A. Yebrin wrote:

> > how about a list of what doesn't build?
> >
> Something like:
>
> r.digit
> r.out.png
> r.out.ppm3
> r.out.tiff
> r.random
> r.region
> r.resamp.rst
> r.surf.fractal
> r.surf.gauss
> r.surf.idw2
> r.surf.random
> r.terraflow
> r.timestamp
> r.to.vect
>
> Some of them will not work because they need a GUI
(r.digit). Others
> maybe because of wrong configure flags (r.out.***).

Can you provide the actual build errors for the above modules?

--
Glynn Clements <glynn@gclements.plus.com>

10

Javier A. Yebrin wrote:

I'll send the build errors as soon I get my work organized. I've started
with some real testing and recompilation. The current module undergoing this
test is r.average, which is presenting some problems because of system calls
to r.stats and r.recode.

Does the Windows runtime not support system(), or is r.average using
Unix-specific features?

I'm thinking of writing the r.stats/recode modules as functions inside
r.average. Do you think this is feasable?

Not in all cases. GRASS modules need to be able to execute other
programs.

I have written some replacements for system() in lib/gis/spawn.c.
However, these haven't been tested yet, and are currently only
implemented for Unix.

Ultimately, all calls to system() should be replaced with an interface
which can reasonably be implemented on all common platforms. For
Windows, this would involve CreateProcess() (or possibly _spawnl(), as
is used in G_system() in lib/gis/system.c).

Even on Unix, system() isn't a particularly robust mechanism for
invoking other programs. Primarily because it uses the shell, which
performs complex translations on the argument.

--
Glynn Clements <glynn@gclements.plus.com>

11

On Fri, May 20, 2005 at 10:48:47PM +0100, Glynn Clements wrote:

Javier A. Yebrin wrote:

> I'll send the build errors as soon I get my work organized. I've started
> with some real testing and recompilation. The current module undergoing this
> test is r.average, which is presenting some problems because of system calls
> to r.stats and r.recode.

Does the Windows runtime not support system(), or is r.average using
Unix-specific features?

A question:
Is it wrong to use G_system() within r.average instead of system()?

--- main.c 15 Jan 2005 11:18:12 -0000 2.1
+++ main.c 18 May 2005 08:16:43 -0000 2.2
@@ -90,7 +90,7 @@
     tempfile2 = G_tempfile();
     sprintf (command, "%s -anC input='%s,%s' fs=space > %s",
        STATS, basemap->answer, covermap->answer, tempfile1);
- if (stat = system(command))
+ if (stat = G_system(command))
     {
        unlink(tempfile1);
        fprintf (stderr, "%s: ERROR running %s command\n", me, STATS);
@@ -151,7 +151,7 @@
     fclose (fd2);
     sprintf (command, "%s input='%s' output='%s' < %s",
        RECODE, basemap->answer, outputmap->answer, tempfile2);
- stat = system(command);
+ stat = G_system(command);
     unlink (tempfile1);
     unlink (tempfile2);
     exit(stat);

?

Markus

12

Markus Neteler wrote:

> > I'll send the build errors as soon I get my work organized. I've started
> > with some real testing and recompilation. The current module undergoing this
> > test is r.average, which is presenting some problems because of system calls
> > to r.stats and r.recode.
>
> Does the Windows runtime not support system(), or is r.average using
> Unix-specific features?

A question:
Is it wrong to use G_system() within r.average instead of system()?

On Unix, G_system() appears to be almost identical to system(), except
that signal handling in the child is slightly different.

system() ignores SIGINT and SIGQUIT in the parent but leaves them
unchanged in the child, while G_system() ignores them in the parent
but restores them to SIG_DFL in the child.

This only makes a difference if the program has changed the way that
those signals are handled from the default behaviour.

So, using G_system() shouldn't affect anything on Unix. I have no idea
whether it will make matters better or worse on Windows.

--
Glynn Clements <glynn@gclements.plus.com>