[GRASS-dev] Re: [GRASS-user] r.in.xyz: Could open text file ~ 2.5GB

GRASS GIS GRASS GIS developer
1

Glynn Clements wrote:

I forgot that the --enable-largefile switch only enables LFS in
libgis, not anywhere else.

[Although config.h includes definitions to enable LFS automatically,
those definitions are currently inactive. This is probably a good
thing; a lot of GRASS' code isn't LFS-aware, and explicit failure is
preferable to silently corrupting data.]

To enable LFS elsewhere, you need to manually add
-D_FILE_OFFSET_BITS=64 to the compilation flags. The simplest approach
is to add:

ifneq ($(USE_LARGEFILES),)
  EXTRA_CFLAGS = -D_FILE_OFFSET_BITS=64
endif

to raster/r.in.xyz/Makefile, then repeat the "make" and "make install"
steps.

I have added this knowledge (and some other tips posted to the mailing
list by Glynn & Brad) to a page in the Development section of the Wiki.
Please correct, update, and expand as needed.

http://grass.gdf-hannover.de/wiki/Large_File_Support

how hard will it be to get a HAVE_FTELLO (or whatever) switch tested &
set by ./configure? If we get that, should every LFS compliant module be
written like:

#ifdef HAVE_FTELLO
    off_t filesize;
#else
    long filesize;
#endif

...

#ifdef HAVE_FTELLO
    fseeko(in_fd, 0L, SEEK_END);
    filesize = ftello(in_fd);
#else
    fseek(in_fd, 0L, SEEK_END);
    filesize = ftell(in_fd);
#endif

Or is that better handled in G_fseek() etc fns etc as suggested by Brad?
(still should try and match variable type in that case (how?))

fopen() and fclose() are ok in both cases? (dependant on finding
-D_FILE_OFFSET_BITS=64 in the Makefile and including grass/config.h ?)

I'm trying to make that wiki page a "best practices" guide for GRASS
developers, so direct updates there would be appreciated.

Hamish

2

Hamish wrote:

> I forgot that the --enable-largefile switch only enables LFS in
> libgis, not anywhere else.
>
> [Although config.h includes definitions to enable LFS automatically,
> those definitions are currently inactive. This is probably a good
> thing; a lot of GRASS' code isn't LFS-aware, and explicit failure is
> preferable to silently corrupting data.]
>
> To enable LFS elsewhere, you need to manually add
> -D_FILE_OFFSET_BITS=64 to the compilation flags. The simplest approach
> is to add:
>
> ifneq ($(USE_LARGEFILES),)
> EXTRA_CFLAGS = -D_FILE_OFFSET_BITS=64
> endif
>
> to raster/r.in.xyz/Makefile, then repeat the "make" and "make install"
> steps.

I have added this knowledge (and some other tips posted to the mailing
list by Glynn & Brad) to a page in the Development section of the Wiki.
Please correct, update, and expand as needed.

http://grass.gdf-hannover.de/wiki/Large_File_Support

how hard will it be to get a HAVE_FTELLO (or whatever) switch tested &
set by ./configure?

The trick is deciding what to test, and what to indicate. Do we want
to know:

1. Whether ftello() exists with the default CFLAGS/CPPFLAGS?

2. Whether ftello() exists with the default CFLAGS/CPPFLAGS plus a
fixed selection of additional switches (e.g. -D_LARGEFILE_SOURCE)?

3. Whether ftello() exists with the default CFLAGS/CPPFLAGS plus a
variable selection of additional switches (e.g. -D_LARGEFILE_SOURCE)
with those switches being communicated via an additional variable?

If we get that, should every LFS compliant module be
written like:

#ifdef HAVE_FTELLO
    off_t filesize;
#else
    long filesize;
#endif

No. You may as well just use "off_t filesize" unconditionally. An
"off_t" will always be large enough to hold a "long".

...

#ifdef HAVE_FTELLO
    fseeko(in_fd, 0L, SEEK_END);
    filesize = ftello(in_fd);
#else
    fseek(in_fd, 0L, SEEK_END);
    filesize = ftell(in_fd);
#endif

The test would need to be:

  #if defined(HAVE_FTELLO) && defined(HAVE_FSEEKO)

as you are using both of those.

Also, if the existence of FSEEKO/FTELLO is conditional upon certain
macros (e.g. _LARGEFILE_SOURCE), you have to ensure that those macros
are defined before the first inclusion of <stdio.h>, including any
"hidden" inclusion from other headers, which essentially means that
you have to ensure that the macros are defined before you include any
other headers (except for <grass/config.h>).

Or is that better handled in G_fseek() etc fns etc as suggested by Brad?
(still should try and match variable type in that case (how?))

My inclination would be to add:

  extern off_t G_ftell(FILE *fp);
  extern int G_fseek(FILE *stream, off_t offset, int whence);

These would be implemented using fseeko/ftello where available, and
fseek/ftell otherwise. That eliminates the need to perform checks in
individual source files. However, we would need to take care not to
use them in code which can't handle large offsets (i.e. code which
will truncate off_t values to int/long).

fopen() and fclose() are ok in both cases? (dependant on finding
-D_FILE_OFFSET_BITS=64 in the Makefile and including grass/config.h ?)

If you use -D_FILE_OFFSET_BITS=64, fopen() should be redirected to
fopen64(). There are reports that the MacOSX fopen() is equivalent to
fopen64() even without additional switches.

--
Glynn Clements <glynn@gclements.plus.com>

3

Glynn Clements wrote:

> If we get that, should every LFS compliant module be
> written like:
>
> #ifdef HAVE_FTELLO
> off_t filesize;
> #else
> long filesize;
> #endif

No. You may as well just use "off_t filesize" unconditionally. An
"off_t" will always be large enough to hold a "long".

will all compilers (within reason) know about the off_t type?

Hamish

4

Hamish wrote:

> > If we get that, should every LFS compliant module be
> > written like:
> >
> > #ifdef HAVE_FTELLO
> > off_t filesize;
> > #else
> > long filesize;
> > #endif
>
> No. You may as well just use "off_t filesize" unconditionally. An
> "off_t" will always be large enough to hold a "long".

will all compilers (within reason) know about the off_t type?

It's a POSIX-ism, although it is provided by the MSVCRT headers:

http://msdn2.microsoft.com/en-us/library/323b6b3k.aspx

Also, a lot of GRASS (including the core raster I/O) will fail to
compile without it.

On some platforms, you may need to explicitly include <sys/types.h>
for off_t to be visible.

--
Glynn Clements <glynn@gclements.plus.com>