[GRASS5] raster filename length problem

GRASS GRASS developer
1

Hi,

there is a problem with long filenames somewhere:

r.reclass.area modis_evi_class_forest out=modis_evi_class_forest_nosmall greater=100
d.rast modis_evi_class_forest_nosmall
WARNING: unable to open [modis_evi_class_forest_nosmall] in [bluetongue]
         since it is a reclass of
         [modis_evi_class_forest.clump.modis_evi_class_foresbluetongue] in
                                                            ^-PROBLEM
         [bluetongue] which does not exist
WARNING: unable to open raster map [modis_evi_class_forest_nosmall in
         bluetongue]
ERROR: Not able to open cellfile for [modis_evi_class_forest_nosmall]

I searched around in r.stats, r.clump, r.reclass, d.rast but didn't
find the problem. The desired filename interally created by r.reclass.area
will be of around 60 chars.

In the source code different name/mapset string length are scattered
around. What about changing the hardcoded ones to a single definition in
include/gis.h?

So far there is
imagery.h:
        #define INAME_LEN 30
        char name[30]; /* length is not in sync with other definitions */
        char mapset[30];

which is definitly too short.

Opinions?

Markus

2

Hi Markus, list,

Markus Neteler wrote:

Hi,

there is a problem with long filenames somewhere:

r.reclass.area modis_evi_class_forest out=modis_evi_class_forest_nosmall greater=100
d.rast modis_evi_class_forest_nosmall
WARNING: unable to open [modis_evi_class_forest_nosmall] in [bluetongue]
        since it is a reclass of
        [modis_evi_class_forest.clump.modis_evi_class_foresbluetongue] in
                                                           ^-PROBLEM
        [bluetongue] which does not exist
WARNING: unable to open raster map [modis_evi_class_forest_nosmall in
        bluetongue]
ERROR: Not able to open cellfile for [modis_evi_class_forest_nosmall]

I searched around in r.stats, r.clump, r.reclass, d.rast but didn't
find the problem. The desired filename interally created by r.reclass.area
will be of around 60 chars.

In the source code different name/mapset string length are scattered
around. What about changing the hardcoded ones to a single definition in
include/gis.h?

So far there is
imagery.h:
       #define INAME_LEN 30
       char name[30]; /* length is not in sync with other definitions */
       char mapset[30];

which is definitly too short.

Opinions?

I encountered more or less the same problem, concerning names of
imaginary groups in grass (cvs version). first I thought setting
INAME_LEN in imagery.h to 256 or something like that would do the trick,
but things appeared a lot more complex than that. Sometimes other
lengths are used (something like 128 if I recall correctly) and often
buffers are allocated not using INAME_LEN but just a fixed integer number.
To sum things up, I started to look at it with the goal to make a simple
patch, but because of the complexity and of other (personal) reasons, I
did never make it, and probably (alas) will not (soon) do so too.

However, when this will get fixed by someone else, I will be most
grateful of course :wink:

Vincent.

Markus

_______________________________________________
grass5 mailing list
grass5@grass.itc.it
http://grass.itc.it/mailman/listinfo/grass5

3

Markus Neteler wrote:

there is a problem with long filenames somewhere:

r.reclass.area modis_evi_class_forest out=modis_evi_class_forest_nosmall greater=100
d.rast modis_evi_class_forest_nosmall
WARNING: unable to open [modis_evi_class_forest_nosmall] in [bluetongue]
         since it is a reclass of
         [modis_evi_class_forest.clump.modis_evi_class_foresbluetongue] in
                                                            ^-PROBLEM
         [bluetongue] which does not exist
WARNING: unable to open raster map [modis_evi_class_forest_nosmall in
         bluetongue]
ERROR: Not able to open cellfile for [modis_evi_class_forest_nosmall]

I searched around in r.stats, r.clump, r.reclass, d.rast but didn't
find the problem. The desired filename interally created by r.reclass.area
will be of around 60 chars.

In gis.h:

  struct Reclass
  {
      char name[50]; /* name of cell file being reclassed */
      char mapset[50]; /* mapset in which "name" is found */
[snip]

More generally, it would help if we had standardised NAME_MAX and
MAPSET_MAX constants, rather than having arbitrary integer literals
scattered all over the code.

--
Glynn Clements <glynn@gclements.plus.com>

4

On Tue, Jul 12, 2005 at 08:05:21PM +0100, Glynn Clements wrote:

Markus Neteler wrote:

> there is a problem with long filenames somewhere:
>
> r.reclass.area modis_evi_class_forest out=modis_evi_class_forest_nosmall greater=100
> d.rast modis_evi_class_forest_nosmall
> WARNING: unable to open [modis_evi_class_forest_nosmall] in [bluetongue]
> since it is a reclass of
> [modis_evi_class_forest.clump.modis_evi_class_foresbluetongue] in
> ^-PROBLEM
> [bluetongue] which does not exist
> WARNING: unable to open raster map [modis_evi_class_forest_nosmall in
> bluetongue]
> ERROR: Not able to open cellfile for [modis_evi_class_forest_nosmall]
>
> I searched around in r.stats, r.clump, r.reclass, d.rast but didn't
> find the problem. The desired filename interally created by r.reclass.area
> will be of around 60 chars.

In gis.h:

  struct Reclass
  {
      char name[50]; /* name of cell file being reclassed */
      char mapset[50]; /* mapset in which "name" is found */
[snip]

Right, that's the beast.

More generally, it would help if we had standardised NAME_MAX and
MAPSET_MAX constants, rather than having arbitrary integer literals
scattered all over the code.

I'll try whenever I come across NAME/MAPSET definitions to set them
to NAME_MAX,MAPSET_MAX. Before adding those to gis.h, how long?
Is 100 ok during the migration to NAME_MAX,MAPSET_MAX? I'm not
sure how to capture string overflows.

Markus

5

> More generally, it would help if we had standardised NAME_MAX and
> MAPSET_MAX constants, rather than having arbitrary integer literals
> scattered all over the code.

I'll try whenever I come across NAME/MAPSET definitions to set them
to NAME_MAX,MAPSET_MAX. Before adding those to gis.h, how long?
Is 100 ok during the migration to NAME_MAX,MAPSET_MAX? I'm not
sure how to capture string overflows.

From memory- Mac OS9 was 40 chars, OSX is unix so fs dependent?; K3B
reports Joliet FS is 103 chars, something is 31 chars, some are 37
(maybe that was where I was getting 40 from for OS9).

So should NAME_MAX,MAPSET_MAX be 128 to not be the limiting factor in
all (reasonable) cases? At least bigger than Joliet?

Does this affect a large MAX_FILES setting much or is it too small a
component to matter?

Hamish

6

On Wed, Jul 13, 2005 at 05:11:25PM +1200, Hamish wrote:

> > More generally, it would help if we had standardised NAME_MAX and
> > MAPSET_MAX constants, rather than having arbitrary integer literals
> > scattered all over the code.
>
> I'll try whenever I come across NAME/MAPSET definitions to set them
> to NAME_MAX,MAPSET_MAX. Before adding those to gis.h, how long?
> Is 100 ok during the migration to NAME_MAX,MAPSET_MAX? I'm not
> sure how to capture string overflows.

From memory- Mac OS9 was 40 chars, OSX is unix so fs dependent?; K3B
reports Joliet FS is 103 chars, something is 31 chars, some are 37
(maybe that was where I was getting 40 from for OS9).

So should NAME_MAX,MAPSET_MAX be 128 to not be the limiting factor in
all (reasonable) cases? At least bigger than Joliet?

Does this affect a large MAX_FILES setting much or is it too small a
component to matter?

I have added
#define GNAME_MAX 50
#define GMAPSET_MAX 50

to include/gis.h and modified lib/gis/ and a couple of modules to
use the global definition. Note that NAME_MAX is already used somewhere
deep in the Linux system includes.

In a similar way I have touched
include/imagery.h
and lib/imagery/ to use INAME_LEN (which is still only 30 chars long).

As a test I have locally changed GNAME_MAX/GMAPSET_MAX to a length of
100 chars and voila', the earlier reported problem of
r.reclass.area/d.rast is solved!

TODO:
- please update from CVS and try longer GNAME_MAX/GMAPSET_MAX/INAME_LEN
- fix overseen hardcoded name/mapset definitions in CVS accordingly
- report success/problems to this list
- once stabilized, we can extend it in CVS.

Markus

7

Markus Neteler wrote:

> More generally, it would help if we had standardised NAME_MAX and
> MAPSET_MAX constants, rather than having arbitrary integer literals
> scattered all over the code.

I'll try whenever I come across NAME/MAPSET definitions to set them
to NAME_MAX,MAPSET_MAX. Before adding those to gis.h, how long?
Is 100 ok during the migration to NAME_MAX,MAPSET_MAX? I'm not
sure how to capture string overflows.

The set_data program limits the location and mapset to 25 characters
each (not including the terminating NUL). I suggest 256 for NAME_MAX
(most filesystems impose a limit of 255 bytes or less for filenames).

I can't think of a reliable way of locating undersized buffers other
than waiting for bug reports.

--
Glynn Clements <glynn@gclements.plus.com>

8

Hamish wrote:

> > More generally, it would help if we had standardised NAME_MAX and
> > MAPSET_MAX constants, rather than having arbitrary integer literals
> > scattered all over the code.
>
> I'll try whenever I come across NAME/MAPSET definitions to set them
> to NAME_MAX,MAPSET_MAX. Before adding those to gis.h, how long?
> Is 100 ok during the migration to NAME_MAX,MAPSET_MAX? I'm not
> sure how to capture string overflows.

>From memory- Mac OS9 was 40 chars, OSX is unix so fs dependent?; K3B
reports Joliet FS is 103 chars, something is 31 chars, some are 37
(maybe that was where I was getting 40 from for OS9).

So should NAME_MAX,MAPSET_MAX be 128 to not be the limiting factor in
all (reasonable) cases? At least bigger than Joliet?

Does this affect a large MAX_FILES setting much or is it too small a
component to matter?

With MAX_FILES=256, 256 chars for a map name equates to 64KB; libgis'
BSS segment is currently 248KB, of which G__ accounts for 150KB, of
which 12KB is the name fields in the Reclass structures.

--
Glynn Clements <glynn@gclements.plus.com>

9

On Thu, Jul 14, 2005 at 02:00:08AM +0100, Glynn Clements wrote:

Markus Neteler wrote:

> > More generally, it would help if we had standardised NAME_MAX and
> > MAPSET_MAX constants, rather than having arbitrary integer literals
> > scattered all over the code.
>
> I'll try whenever I come across NAME/MAPSET definitions to set them
> to NAME_MAX,MAPSET_MAX. Before adding those to gis.h, how long?
> Is 100 ok during the migration to NAME_MAX,MAPSET_MAX? I'm not
> sure how to capture string overflows.

The set_data program limits the location and mapset to 25 characters
each (not including the terminating NUL).

In lib/init/set_data.c we have

    char gisdbase[70];
    char location_name[41] ;
    char location[1024] ;
    char mapset[41] ;

        V_ques (location_name, 's', line++, 12, 25);
        ...
        V_ques (mapset, 's', line++, 12, 25);
        ...
        V_ques (gisdbase, 's', line++, 10, sizeof(gisdbase) - 1);

Which looks a bit messy to me.
Is there a problem to set things to GNAME_NAME length?

I suggest 256 for NAME_MAX
(most filesystems impose a limit of 255 bytes or less for filenames).

I can't think of a reliable way of locating undersized buffers other
than waiting for bug reports.

For me that's fine. I'll change locally to this value to see what happens.
With a few scripts things could be tested automatically (use g.proj to
generate new locations with long names in different projections;
use r.mapcalc or r.surf.fractal to generate landscapes etc).

Markus

10

Markus Neteler wrote:

> > > More generally, it would help if we had standardised NAME_MAX and
> > > MAPSET_MAX constants, rather than having arbitrary integer literals
> > > scattered all over the code.
> >
> > I'll try whenever I come across NAME/MAPSET definitions to set them
> > to NAME_MAX,MAPSET_MAX. Before adding those to gis.h, how long?
> > Is 100 ok during the migration to NAME_MAX,MAPSET_MAX? I'm not
> > sure how to capture string overflows.
>
> The set_data program limits the location and mapset to 25 characters
> each (not including the terminating NUL).

In lib/init/set_data.c we have

    char gisdbase[70];
    char location_name[41] ;
    char location[1024] ;
    char mapset[41] ;

        V_ques (location_name, 's', line++, 12, 25);
        ...
        V_ques (mapset, 's', line++, 12, 25);
        ...
        V_ques (gisdbase, 's', line++, 10, sizeof(gisdbase) - 1);

Which looks a bit messy to me.
Is there a problem to set things to GNAME_NAME length?

Presumably you mean GMAPSET_MAX? There isn't a map name field there.
The location should probably have a similar constant, although that's
less critical as so few modules explicitly deal with locations.

     char mapset[GMAPSET_MAX];

         V_ques (mapset, 's', line++, 12, sizeof(mapset) - 1);

I'm not sure what will happen if GMAPSET_MAX is set too high (i.e.
whether the field will go off of the screen; it depends upon whether
the vask library has a check for this).

--
Glynn Clements <glynn@gclements.plus.com>