Been a while since I have used GRASS, and does it feel good to be back using…
Am using 5.4, the latest compiled for Linux…
Trying to find the best way to approach combining ~4400 raster images into one. Description are 50 foot ESRI Ascii formatted DEMs imported using r.in.arc. When attempting to use a bash script to combine these rasters using r.mapcalc, the error appears “ERROR: Too many open raster files”
Does anyone know the limit of number of rasters that can be opened at one time? Or does anyone have a suggestion on how best to combine (do I use r.combine??) the 4400 rasters as painless as possible?
I do not recall the exact file, but a constant in the source code defining the
max_open_files or something like that needs to be changed. Check the list's
archives for a post by Hamish
Cheers,
Dylan
On Thursday 20 July 2006 11:27, Slover, Kevin wrote:
Been a while since I have used GRASS, and does it feel good to be back
using...
Am using 5.4, the latest compiled for Linux...
Trying to find the best way to approach combining ~4400 raster images
into one. Description are 50 foot ESRI Ascii formatted DEMs imported
using r.in.arc. When attempting to use a bash script to combine these
rasters using r.mapcalc, the error appears "ERROR: Too many open raster
files"
Does anyone know the limit of number of rasters that can be opened at
one time? Or does anyone have a suggestion on how best to combine (do I
use r.combine??) the 4400 rasters as painless as possible?
Thanks!!
Kevin Slover
Coastal / GIS Specialist
2872 Woodcock Blvd Suite 230
Atlanta GA 30341
(P) 678-530-0022
(F) 678-530-0044
--
Dylan Beaudette
Soils and Biogeochemistry Graduate Group
University of California at Davis
530.754.7341
On 7/20/06, Dylan Beaudette <dylan.beaudette@gmail.com> wrote:
Hi,
I do not recall the exact file, but a constant in the source code defining the
max_open_files or something like that needs to be changed. Check the list's
archives for a post by Hamish
Cheers,
Dylan
On Thursday 20 July 2006 11:27, Slover, Kevin wrote:
> Been a while since I have used GRASS, and does it feel good to be back
> using...
>
> Am using 5.4, the latest compiled for Linux...
>
> Trying to find the best way to approach combining ~4400 raster images
> into one. Description are 50 foot ESRI Ascii formatted DEMs imported
> using r.in.arc. When attempting to use a bash script to combine these
> rasters using r.mapcalc, the error appears "ERROR: Too many open raster
> files"
>
> Does anyone know the limit of number of rasters that can be opened at
> one time? Or does anyone have a suggestion on how best to combine (do I
> use r.combine??) the 4400 rasters as painless as possible?
>
> Thanks!!
>
> Kevin Slover
>
> Coastal / GIS Specialist
>
> 2872 Woodcock Blvd Suite 230
>
> Atlanta GA 30341
>
> (P) 678-530-0022
>
> (F) 678-530-0044
--
Dylan Beaudette
Soils and Biogeochemistry Graduate Group
University of California at Davis
530.754.7341
--
+-----------------------------------------------------------+
Carlos Henrique Grohmann - Guano
Geologist M.Sc - Doctorate Student at IGc-USP - Brazil
Linux User #89721 - carlos dot grohmann at gmail dot com
+-----------------------------------------------------------+
_________________
"Good morning, doctors. I have taken the liberty of removing Windows
95 from my hard drive."
--The winning entry in a "What were HAL's first words" contest judged
by 2001: A SPACE ODYSSEY creator Arthur C. Clarke
I do not recall the exact file, but a constant in the source code
defining the max_open_files or something like that needs to be
changed.
grass6/raster/r.patch/nfiles.h :
/* The number of cell files that can be patched together.
*
* All cell files will be opened at one time, so this number can not
* be arbitrarily large.
*
* Must be smaller than MAXFILES as defined in lib/gis/G.h which
* in turn must be smaller than the operating system's limit.
* (Given by `cat /proc/sys/fs/file-max` in Linux 2.4)
*/
#define MAXFILES 200
(this is the same in GRASS 5.4.0)
you can increase those numbers (a bit) or run the task in smaller
sections, then combine the pieces with another r.patch pass as a final
step.
tip:
INMAPS=map1,map2,map3,map4,... # or `g.mlist sep=,`
g.region rast=$INMAPS # extend region to extent of all maps
Trying to find the best way to approach combining ~4400 raster images
into one. Description are 50 foot ESRI Ascii formatted DEMs imported
using r.in.arc. When attempting to use a bash script to combine these
rasters using r.mapcalc, the error appears "ERROR: Too many open raster
files"
Does anyone know the limit of number of rasters that can be opened at
one time? Or does anyone have a suggestion on how best to combine (do I
use r.combine??) the 4400 rasters as painless as possible?
You need to recompile to change the upper limit; both libgis and
r.patch have their own limits. Even if you do that, 4400 open files
may exceed limits imposed by the OS.
For that many files, I would suggest a recursive approach, i.e.
combine 440 batches of 100 rasters to give 440 files, combine batches
of those files together, etc.
On Fri, Jul 21, 2006 at 11:20:40AM +0100, Glynn Clements wrote:
...
You need to recompile to change the upper limit; both libgis and
r.patch have their own limits. Even if you do that, 4400 open files
may exceed limits imposed by the OS.
Curiosity:
Would it makes sense to unify these different limits into
a single one (gis.h or so)? Also xganim is affected and others.
From a user perspective it would be less confusing.
On Friday 21 July 2006 15:03, Markus Neteler wrote:
On Fri, Jul 21, 2006 at 11:20:40AM +0100, Glynn Clements wrote:
...
> You need to recompile to change the upper limit; both libgis and
> r.patch have their own limits. Even if you do that, 4400 open files
> may exceed limits imposed by the OS.
Curiosity:
Would it makes sense to unify these different limits into
a single one (gis.h or so)? Also xganim is affected and others.
>From a user perspective it would be less confusing.
Markus
I think that it would be a good thing, as tracking down all of these changes
can be time consuming if you don't do it on a regular basis.
Cheers,
--
Dylan Beaudette
Soils and Biogeochemistry Graduate Group
University of California at Davis
530.754.7341
> You need to recompile to change the upper limit; both libgis and
> r.patch have their own limits. Even if you do that, 4400 open files
> may exceed limits imposed by the OS.
Curiosity:
Would it makes sense to unify these different limits into
a single one (gis.h or so)? Also xganim is affected and others.
From a user perspective it would be less confusing.
It would be better to just do away with the limits, i.e. change
G__.fileinfo from an array to a pointer and dynamically re-allocate it
as necessary. Similarly for individual modules.
On Sat, Jul 22, 2006 at 12:38:58AM +0100, Glynn Clements wrote:
Markus Neteler wrote:
> > You need to recompile to change the upper limit; both libgis and
> > r.patch have their own limits. Even if you do that, 4400 open files
> > may exceed limits imposed by the OS.
>
> Curiosity:
> Would it makes sense to unify these different limits into
> a single one (gis.h or so)? Also xganim is affected and others.
> From a user perspective it would be less confusing.
It would be better to just do away with the limits, i.e. change
G__.fileinfo from an array to a pointer and dynamically re-allocate it
as necessary. Similarly for individual modules.
Sounds good but it's beyond my C knowledge.
So let's hope for a volunteer.
> > > You need to recompile to change the upper limit; both libgis and
> > > r.patch have their own limits. Even if you do that, 4400 open files
> > > may exceed limits imposed by the OS.
> >
> > Curiosity:
> > Would it makes sense to unify these different limits into
> > a single one (gis.h or so)? Also xganim is affected and others.
> > From a user perspective it would be less confusing.
>
> It would be better to just do away with the limits, i.e. change
> G__.fileinfo from an array to a pointer and dynamically re-allocate it
> as necessary. Similarly for individual modules.
Sounds good but it's beyond my C knowledge.
So let's hope for a volunteer.
I've changed this in the CVS head. As it's potentially destabilising,
I wouldn't recommend backporting it to 6.1.