Hi,
while trying to export a few hundred maps into a
big ERDAS/IMG multichannel file (using the GDAL/GRASS plugin
and gdal_merge.py), I stumbled over
lib/gis/G.h :#define MAXFILES 256
While this number is lower than 365 days, Radim
reminded me of the memory allocation.
How to deal with this problem? Simply changing
MAXFILES to a higher number will cause problems
on handhelds.
Markus
while trying to export a few hundred maps into a
big ERDAS/IMG multichannel file (using the GDAL/GRASS plugin
and gdal_merge.py), I stumbled overlib/gis/G.h :#define MAXFILES 256
While this number is lower than 365 days, Radim
reminded me of the memory allocation.How to deal with this problem? Simply changing
MAXFILES to a higher number will cause problems
on handhelds.
Low memory or stripped down kernel maxfiles?
FWIW, boosting G.h to 400 (ie >365) for use with r.series works fine
for me.
see also the /* comments */ in here:
http://freegis.org/cgi-bin/viewcvs.cgi/grass6/raster/r.patch/nfiles.h
there was some talk about this on the mailing list; maybe about the time
of rev1.4, Jul 2003?
Hamish
On Fri, Mar 25, 2005 at 03:10:39PM +1200, Hamish wrote:
> while trying to export a few hundred maps into a
> big ERDAS/IMG multichannel file (using the GDAL/GRASS plugin
> and gdal_merge.py), I stumbled over
>
> lib/gis/G.h :#define MAXFILES 256
>
> While this number is lower than 365 days, Radim
> reminded me of the memory allocation.
>
> How to deal with this problem? Simply changing
> MAXFILES to a higher number will cause problems
> on handhelds.Low memory or stripped down kernel maxfiles?
No, it's sort of unrelated to kernel maxfiles.
The point is that in lib/gis/G.h the fileinfo struct
is allocated as fileinfo[MAXFILES]. So all the contents
of this struct are allocated as often as MAXFILES is
defined.
FWIW, boosting G.h to 400 (ie >365) for use with r.series works fine
for me.
I tried 1024, no problem. But: in this case we have "lost"
struct fileinfo[1024] memory allocation.
In this struct fileinfo are
- 10 ints : 4 * 10 = 40
- 1 double : 8 * 8 = 64
- 4 chars : 1 * 4 = 4
- 3 unsigned chars : 1 * 3 = 3
- 6 further structs:
- Cell_head (include/gis.h):
- 9 ints : 4 * 9 = 36
- 11 doubles : 8 * 11 = 88
- Reclass:
- 2 chars : 1 * 2 = 2
- 2 ints : 4 * 2 = 8
- 3 CELL : 4 * 3 = 12
- Cell_stats
- struct Cell_stats_node
- 3 ints : 4 * 3 = 12
- 1 long : 4 * 1 = 4 (or 8 * 1 = 8)
- 4 ints : 4 * 4 = 16
- 1 long : 4 * 1 = 4
- Range
- 1 int : 4 * 1 = 4
- 2 CELL : 4 * 2 = 8
- FPRange
- 1 int : 4 * 1 = 4
- 2 DCELL : 8 * 2 = 16
- Quant
- 9 ints : 4 * 9 = 36
- 6 CELLs : 4 * 6 = 24
- 6 DCELLs : 8 * 6 = 48
- struct Quant_table
- 2 CELLs : 4 * 2 = 8
- 2 DCELLs : 8 * 2 = 16
- struct fp_lookup
- 2 ints : 4 * 2 = 8
- 2 CELLs : 4 * 2 = 8
- 2 DCELLs : 8 * 2 = 16
- struct Quant_table
- 2 CELLs : 4 * 2 = 8
- 2 DCELLs : 8 * 2 = 16
- 1 off_t : 4 * 1 = 4
- 1 RASTER_MAP_TYPE: typedef int : 4 * 1 = 4
- 1 COLUMN_MAPPING: typedef int : 4 * 1 = 4
- 1 XDR: register/typedef struct : ? (/usr/include/rpc/xdr.h)
======================================================
SUM 525 bytes + XDR
So we would allocate 525 bytes * 1024 = 537600 (+ XDR!)
When running GRASS, around 0.5MB of RAM are immediately consumed.
It's likely that I made some mistakes but I just wanted
to estimate it.
Using
fprintf(stderr,"sizeof fileinfo: %d\n", sizeof(struct fileinfo));
prints 600 bytes. This number then multiplied by MAXFILES...
So it becomes quite memory consuming for handhelds.
Markus
Markus Neteler wrote:
> > while trying to export a few hundred maps into a
> > big ERDAS/IMG multichannel file (using the GDAL/GRASS plugin
> > and gdal_merge.py), I stumbled over
> >
> > lib/gis/G.h :#define MAXFILES 256
> >
> > While this number is lower than 365 days, Radim
> > reminded me of the memory allocation.
> >
> > How to deal with this problem? Simply changing
> > MAXFILES to a higher number will cause problems
> > on handhelds.
>
> Low memory or stripped down kernel maxfiles?No, it's sort of unrelated to kernel maxfiles.
The point is that in lib/gis/G.h the fileinfo struct
is allocated as fileinfo[MAXFILES]. So all the contents
of this struct are allocated as often as MAXFILES is
defined.
You could change it to "struct fileinfo *" and reallocate the array
dynamically. However, you would need to be careful about holding
pointers to the individual slots, as they may be relocated if the
array is resized.
--
Glynn Clements <glynn@gclements.plus.com>