[GRASS-user] r.patch: GRASS-6.5svn

GRASS GRASS user
1

   I have 9 raster elevation files that cover the entire state. When displayed
simultaneously there are obvious mis-matches along the edges apparently
because of different hight ranges. Yesterday I attempted to create a single
DEM map for the entire state using r.patch.

   These are the steps I followed:

   MAPS=`g.mlist type=rast sep=, pat="dem*"`
   g.region rast=$MAPS
   r.patch in=$MAPS out=demOR

Then I went off and did other things for a couple of hours.

   There was a warning about inability to write line 24nnn (I forget the
exact line number), and when it was finished and I tried to display the
whole new map, only the top portion (I'd estimate the top half) was present.

   Should I do this in slices (3 maps at a time), then patch the results into
a single map? What might cause the results I saw?

Rich

2

On Fri, 15 Jan 2010, Rich Shepard wrote:

There was a warning about inability to write line 24nnn (I forget the
exact line number), and when it was finished and I tried to display the
whole new map, only the top portion (I'd estimate the top half) was present.

   I need ideas on how to find the source of the problem.

   Made 3 horizontal maps, north, central, south from the 9 maps. No problems
making or displaying them. However, ...

   When I tried to patch the three together the error displayed 'unable to
write row 24580.' I removed that map attempt and tried to patch the north
and central maps together; this produced the error, 'unable to write row
24581.'

   What might cause such an error? This is in the central band which displays
without comment by itself.

Rich

3

Rich Shepard wrote:

> There was a warning about inability to write line 24nnn (I forget the
> exact line number), and when it was finished and I tried to display the
> whole new map, only the top portion (I'd estimate the top half) was present.

   I need ideas on how to find the source of the problem.

   Made 3 horizontal maps, north, central, south from the 9 maps. No problems
making or displaying them. However, ...

   When I tried to patch the three together the error displayed 'unable to
write row 24580.' I removed that map attempt and tried to patch the north
and central maps together; this produced the error, 'unable to write row
24581.'

   What might cause such an error? This is in the central band which displays
without comment by itself.

The most likely reasons are:

1. Your version of GRASS was built without LFS (large file support),
and the cell/fcell file reached the 2GiB limit.

2. The disk is full or you exceeded your quota.

If you're building GRASS yourself, be sure to use the
--enable-largefile option when running configure.

--
Glynn Clements <glynn@gclements.plus.com>

4

On Fri, 15 Jan 2010, Glynn Clements wrote:

The most likely reasons are:

1. Your version of GRASS was built without LFS (large file support),
and the cell/fcell file reached the 2GiB limit.

Glynn,

   Nope. LFS was enabled.

2. The disk is full or you exceeded your quota.

   Nope. There are 14G available on that partition.

If you're building GRASS yourself, be sure to use the enable-largefile
--option when running configure.

   See 1 above.

   I've been experimenting today and I think it's an error in one of the 9
individual maps. Probably one of those in the middle. I get the same error
when I try to patch the top third or bottom third with the middle third.

   Now I'll patch them together one map at a time until I identify the
culprit, then I'll re-import that puppy.

Rich

5

On Fri, 15 Jan 2010, Rich Shepard wrote:

I need ideas on how to find the source of the problem. What might cause
such an error?

   Does anyone know why r.patch and/or r.series would generate an error about
not being able to write a row when maps are concatenated? It seems to me
that if the data are good (which they seem to be), and I can combine the two
suspect maps (of the 9 total) with surrounding ones one at a time until a
threshold is reached, it must be something with the two modules that's
failing.

   I'd really like to fix this so I can get the DEM for the entire state
before I try for other states.

Thanks,

Rich

6

Rich Shepard wrote:

> I need ideas on how to find the source of the problem. What might cause
> such an error?

   Does anyone know why r.patch and/or r.series would generate an error about
not being able to write a row when maps are concatenated? It seems to me
that if the data are good (which they seem to be), and I can combine the two
suspect maps (of the 9 total) with surrounding ones one at a time until a
threshold is reached, it must be something with the two modules that's
failing.

The error "map [%s] - unable to write row %d" doesn't originate in a
module, it originates in the library, and invariably indicates that
write() failed.

[If you're getting a different error, please post the *exact* error
message, not a paraphrase.]

This can be caused by many things, including (but not limited to):

* Exceeding the 2GiB limit on a build without LFS.
* Exceeding the 4GiB limit of a FAT filesystem.
* Exceeding "ulimit -f".
* Exceeding quota.
* Disk full.
* Hardware error

If you built GRASS from source, apply the attached patch with:

  patch -p0 < write_errno.patch

from the top-level directory of the GRASS source tree, then
re-compile.

This will cause the underlying OS error to be included in the error
message.

The issue isn't limited to a specific module. I would expect to see
the same issue with almost any raster command run with the same
region, e.g.:

  r.mapcalc 'test = rand(-1.0,1.0)'

Note: I wouldn't expect to see the issue with a trivial test case such
as:

  r.mapcalc 'test = 0'

because such a map will compress very well, and it's the size of the
compressed data which matters.

--
Glynn Clements <glynn@gclements.plus.com>

(attachments)

write_errno.patch (613 Bytes)

7

On Thu, 21 Jan 2010, Glynn Clements wrote:

The error "map [%s] - unable to write row %d" doesn't originate in a
module, it originates in the library, and invariably indicates that
write() failed.

[If you're getting a different error, please post the *exact* error
message, not a paraphrase.]

Hi Glynn,

   I killed the r.patch process 66% through because the warning was issued
and in previous attempts this stopped the map being completely displayed:

GRASS 6.5.svn (Oregon):/usr4/grassbase > MAPS=`g.mlist type=rast sep=,
pat="demCEN,demE,demN,demNE,demNW,demS,demSE,demSW,demW"`
GRASS 6.5.svn (Oregon):/usr4/grassbase > g.region rast=$MAPS GRASS 6.5.svn (Oregon):/usr4/grassbase > r.patch in=$MAPS out=demOR --o
WARNING: map [demOR] - unable to write row 24578

This can be caused by many things, including (but not limited to):

* Exceeding the 2GiB limit on a build without LFS.

   Here are the configure options I use with -6.4 and -6.5:

--with-postgres \
--with-sqlite \
--with-cairo \
--with-freetype \
--with-cxx \
--with-geos=usr/bin/geos-config \
--with-gdal=/usr/bin/gdal-config \
--with-python=/usr/bin/python-config \
--with-wxwidgets=/usr/bin/wx-config \
--with-includes=/usr/include/ \
--with-libs=/usr/lib/ \
--enable-largefile

* Exceeding the 4GiB limit of a FAT filesystem.

   No FAT on linux.

* Exceeding "ulimit -f".

[rshepard@salmo ~]$ ulimit -f
unlimited

* Exceeding quota.

   Quotas not enabled; or at default values.

* Disk full.

[rshepard@salmo ~]$ df -h /usr4
Filesystem Size Used Avail Use% Mounted on
/dev/hdc3 38G 24G 15G 63% /usr4

* Hardware error

   Perhaps, but no problems elsewhere.

If you built GRASS from source, apply the attached patch with:
  patch -p0 < write_errno.patch
from the top-level directory of the GRASS source tree, then re-compile.

This will cause the underlying OS error to be included in the error
message.

   Here are the results:

GRASS 6.5.svn (Oregon):/usr4/grassbase > MAPS=a.mlist type=rast sep=,
pat="demCEN,demE,demN,demNE,demNW,demS,demSE,demSW,demW"
GRASS 6.5.svn
(Oregon):/usr4/grassbase > g.region rast=$MAPS GRASS 6.5.svn (Oregon):/usr4/grassbase > r.patch in=$MAPS out=demOR --o
WARNING: map [demOR] - unable to write row 24578 (No such file or
          directory)
  100%
Creating support files for raster map <demOR>

and

GRASS 6.5.svn (Oregon):/usr4/grassbase > g.list rast
----------------------------------------------
raster files available in mapset <PERMANENT>:
demCEN demE demN demNE demNW demOR demS demSE demSW demW

so the file is present. But, ...

GRASS 6.5.svn (Oregon):/usr4/grassbase > d.rast demOR
WARNING: Error reading compressed map <demOR@PERMANENT>, row 0
Segmentation fault

   Here's what r.info has to tell about the map:

GRASS 6.5.svn (Oregon):/usr4/grassbase > r.info demOR
+----------------------------------------------------------------------------+
| Layer: demOR Date: Thu Jan 21 18:36:57 2010 |
| Mapset: PERMANENT Login of Creator: rshepard |
| Location: Oregon |
| DataBase: /usr4/grassbase |
| Title: DEM 10m Central Oregon ( demOR ) |
| Timestamp: none |
|----------------------------------------------------------------------------|
| |
| Type of Map: raster Number of Categories: 11231 |
| Data Type: CELL |
| Rows: 53388 |
| Columns: 71949 |
| Total Cells: 3841213212 |
| Projection: Lambert Conformal Conic |
| N: 1737497.65432757 S: -13512.48800518 Res: 32.7978224 |
| E: 2502682.81668463 W: 142938.14294121 Res: 32.79746312 |
| Range of data: min = 0 max = 11231 |

Rich

8

Rich Shepard wrote:

GRASS 6.5.svn (Oregon):/usr4/grassbase > r.patch in=$MAPS out=demOR --o
WARNING: map [demOR] - unable to write row 24578 (No such file or
          directory)

"No such file or directory" is ENOENT, which isn't among the errors
which write() can return. That suggests that write() is succeeding,
but returning a short count (i.e. it wrote some data, not all of it).
I suspect that a subsequent write() will result in an actual error and
indicate the cause, but write_error() only reports the first error (so
that you don't get thousands of them).

Can you revert the previous patch with:

  patch -R -p0 < write_errno.patch

then apply the attached patch with:

  patch -R -p0 < write_errors.patch

and re-compile.

That should cause it to print more than one error.

Also, can you provide the output from:

  ls -l /usr4/grassbase/Oregon/PERMANENT/cell/demOR

BTW, do you get similar errors with other commands using the same
region, e.g.:

  r.mapcalc 'test = rand(9999)'
?

--
Glynn Clements <glynn@gclements.plus.com>

(attachments)

write_errors.patch (733 Bytes)

9

On Fri, 22 Jan 2010, Glynn Clements wrote:

Can you revert the previous patch with:
  patch -R -p0 < write_errno.patch
then apply the attached patch with:
  patch -R -p0 < write_errors.patch
and re-compile.
That should cause it to print more than one error.

GRASS 6.5.svn (Oregon):/usr4/grassbase > r.patch in=$MAPS out=demOR --o WARNING: map [demOR] - unable to write row 24578 (No such file or
          directory)
WARNING: map [demOR] - unable to write row 24578 (File too large)
  100%
Creating support files for raster map <demOR>

   I've not before encountered 'file too large' ...

Also, can you provide the output from:
  ls -l /usr4/grassbase/Oregon/PERMANENT/cell/demOR

   But this explains it:

  2147483647 Jan 22 07:39 /usr4/grassbase/Oregon/PERMANENT/cell/demOR

   That's a 2G file size for 10m resolution DEM of the entire state. Wow!

   What's the best strategy for decreasing file size? Do I not make a
state-wide DEM but make 2 or three instead? Should I keep the 9 separate
maps and concatenate as necessary for specific projects?

   This has been a _very_ useful lesson.

BTW, do you get similar errors with other commands using the same
region, e.g.:
  r.mapcalc 'test = rand(9999)'

GRASS 6.5.svn (Oregon):/usr4/grassbase > r.mapcalc 'test = rand(9999)'
Too few arguments (1) to function rand()
Parse error

Thanks,

Rich

10

Rich Shepard wrote:

> Can you revert the previous patch with:
> patch -R -p0 < write_errno.patch
> then apply the attached patch with:
> patch -R -p0 < write_errors.patch
> and re-compile.
> That should cause it to print more than one error.

GRASS 6.5.svn (Oregon):/usr4/grassbase > r.patch in=$MAPS out=demOR --o
WARNING: map [demOR] - unable to write row 24578 (No such file or directory)
WARNING: map [demOR] - unable to write row 24578 (File too large)
WARNING: map [demOR] - unable to write row 24578 (File too large)
  100%
Creating support files for raster map <demOR>

   I've not before encountered 'file too large' ...

The code is EFBIG.

       EFBIG An attempt was made to write a file that exceeds the implementa-
        tion-defined maximum file size or the process's file size limit,
        or to write at a position past the maximum allowed offset.

That rules out disk full (which fails with ENOSPC).

> Also, can you provide the output from:
> ls -l /usr4/grassbase/Oregon/PERMANENT/cell/demOR

   But this explains it:

  2147483647 Jan 22 07:39 /usr4/grassbase/Oregon/PERMANENT/cell/demOR

Yep, that's the 2GiB limit.

   That's a 2G file size for 10m resolution DEM of the entire state. Wow!

   What's the best strategy for decreasing file size? Do I not make a
state-wide DEM but make 2 or three instead? Should I keep the 9 separate
maps and concatenate as necessary for specific projects?

Actually, I want to know why the 2GiB limit is there.

1. Can you confirm that include/Make/Platform.make contains:

  #Large File Support (LFS)
  USE_LARGEFILES = 1

2. Can you re-compile lib/gis and check that the compilation commands
include the flag -D_FILE_OFFSET_BITS=64.

[r.patch shouldn't need any particular compilation flags, as the file
in question is created and written by lib/gis.]

3. Can you provide the files produced by the following commands (run
from within a GRASS session):

  ldd $GIBASE/bin/r.patch > ldd.txt

  nm -D $GISBASE/lib/libgrass_gis.so > nm.txt

   This has been a _very_ useful lesson.

> BTW, do you get similar errors with other commands using the same
> region, e.g.:
> r.mapcalc 'test = rand(9999)'

GRASS 6.5.svn (Oregon):/usr4/grassbase > r.mapcalc 'test = rand(9999)'
Too few arguments (1) to function rand()
Parse error

Oops; that should have been rand(0,9999). But I think I already know
how that would have turned out.

--
Glynn Clements <glynn@gclements.plus.com>

11

On Sat, 23 Jan 2010, Glynn Clements wrote:

Actually, I want to know why the 2GiB limit is there.

Glynn,

   Me, too!

1. Can you confirm that include/Make/Platform.make contains:

  #Large File Support (LFS)
  USE_LARGEFILES = 1

   Yes, those two lines are present.

2. Can you re-compile lib/gis and check that the compilation commands
include the flag -D_FILE_OFFSET_BITS=64.

   lib/gis/Makefile contains:

#compile if LFS Large File Support present:
ifneq ($(USE_LARGEFILES),)
         EXTRA_CFLAGS = -D_FILE_OFFSET_BITS=64
endif

[r.patch shouldn't need any particular compilation flags, as the file
in question is created and written by lib/gis.]

3. Can you provide the files produced by the following commands (run
from within a GRASS session):

  ldd $GIBASE/bin/r.patch > ldd.txt

   We actually want the environmental variable to be $GISBASE.

ldd.txt:

   linux-gate.so.1 => (0xffffe000)
   /lib/libsafe.so.2 (0xb7f77000)
   libgrass_gis.so => /usr/local/grass-6.5.svn/lib/libgrass_gis.so (0xb7f2b000)
   libgrass_datetime.so => /usr/local/grass-6.5.svn/lib/libgrass_datetime.so (0xb7f23000)
   libz.so.1 => /usr/lib/libz.so.1 (0xb7eee000)
   libm.so.6 => /lib/libm.so.6 (0xb7ec8000)
   libc.so.6 => /lib/libc.so.6 (0xb7d7c000)
   libdl.so.2 => /lib/libdl.so.2 (0xb7d78000)
   /lib/ld-linux.so.2 (0xb7f7e000)

  nm -D $GISBASE/lib/libgrass_gis.so > nm.txt

   That file's 905 lines long so I'll attach it to the message rather than
inserting it here.

Thanks,

Rich

(attachments)

nm.txt (25.1 KB)

12

Rich Shepard wrote:

> 2. Can you re-compile lib/gis and check that the compilation commands
> include the flag -D_FILE_OFFSET_BITS=64.

   lib/gis/Makefile contains:

#compile if LFS Large File Support present:
ifneq ($(USE_LARGEFILES),)
         EXTRA_CFLAGS = -D_FILE_OFFSET_BITS=64
endif

Right. But if you do e.g.:

  touch lib/gis/opencell.c
  make -C lib/gis

does the -D_FILE_OFFSET_BITS=64 switch appear in the output?

> [r.patch shouldn't need any particular compilation flags, as the file
> in question is created and written by lib/gis.]
>
> 3. Can you provide the files produced by the following commands (run
> from within a GRASS session):

> nm -D $GISBASE/lib/libgrass_gis.so > nm.txt

   That file's 905 lines long so I'll attach it to the message rather than
inserting it here.

This isn't good:

         U __lxstat

         U __xstat

         U creat

         U fopen

         U freopen

         U lseek
         U lseek64

         U open
         U open64

         U readdir

This implies that most of the library was built without LFS. If I run
the above command here, the corresponding entries are:

         U __lxstat64
         U __xstat64
         U creat64
         U fopen64
         U freopen64
         U lseek64
         U open64
         U readdir64

One possibility is that GRASS was initially built without LFS, then
configure was re-run with --enable-largefile and GRASS was built
without first performing "make clean". This would result in most of
the files not be re-compiled.

I suggest removing both the installed version of GRASS and the source
tree, then compiling from scratch (run make with e.g.
"make &> build.log" to capture all of the output).

After compiling, check the output from "nm ... libgrass_gis.so" and
ensure that all of the above symbols have the "64" suffix. If they
don't, send me the complete make output off-list (compress it first,
as it's likely to be around 3MB uncompressed vs 120KB compressed).

--
Glynn Clements <glynn@gclements.plus.com>

13

On Sun, 24 Jan 2010, Glynn Clements wrote:

Right. But if you do e.g.:

  touch lib/gis/opencell.c
  make -C lib/gis

does the -D_FILE_OFFSET_BITS=64 switch appear in the output?

Glynn,

   It appears so:

gcc -I/usr4/grass65/dist.i686-pc-linux-gnu/include -I/usr/include/ -g -O2
-I/usr/include/ -fPIC -DPACKAGE=\""grasslibs"\" -D_FILE_OFFSET_BITS=64
-DGDAL_LINK=1 -DGDAL_DYNAMIC=1 -DPACKAGE=\""grasslibs"\" -I/usr/include
-I/usr/include -I/usr4/grass65/dist.i686-pc-linux-gnu/include -o
OBJ.i686-pc-linux-gnu/opencell.o -c opencell.c

One possibility is that GRASS was initially built without LFS, then
configure was re-run with --enable-largefile and GRASS was built without
first performing "make clean". This would result in most of the files not
be re-compiled.

I suggest removing both the installed version of GRASS and the source
tree, then compiling from scratch (run make with e.g. "make &> build.log"
to capture all of the output).

   OK.

After compiling, check the output from "nm ... libgrass_gis.so" and ensure
that all of the above symbols have the "64" suffix. If they don't, send me
the complete make output off-list (compress it first, as it's likely to be
around 3MB uncompressed vs 120KB compressed).

   More after I do this.

Rich

14

On Sun, 24 Jan 2010, Glynn Clements wrote:

One possibility is that GRASS was initially built without LFS, then
configure was re-run with --enable-largefile and GRASS was built without
first performing "make clean". This would result in most of the files not
be re-compiled.

Glynn,

   This must have been the case. I ran "make dist-clean", then "configure &
make & make install". When I ran r.patch on the 9 DEM maps it successfully
produced a 4.6G DEM file of the entire state.

   Normally, when I rebuild from source I run "make clean" or "make
dist-clean" first. Why I did not do so this time I don't know. Brain fart,
perhaps.

Thanks for guiding me to the proper solution,

Rich