[GRASS-dev] Maximum safe map name size?

Joel_Pitt · June 30, 2010, 9:54pm

Hi all,

So I've been getting into all sorts of strife with hard-coded limits
on filename lengths in GRASS. Not hard-coded in a sense that they are
enforced (that wouldn't be an issue, because then I'd know about it!),
but in the sense that there are all these fixed buffer limits in
modules which don't gracefully handle filenames that get too long.
Instead it fails with cryptic messages, and since I've built a large
simulation framework/pipeline around GRASS, it makes everything else
fall over too.

I've been having trouble with r.random and r.reclass:

http://trac.osgeo.org/grass/ticket/800

I "fixed" those, but now the cell header for the reclassed map results in:

WARNING: Unable to open raster map

whenever I try to access it. Except, the header in cellhd looks
exactly the same as another file that's reclassed and works fine. The
only difference is that the working one reclasses a map with a short
name.

I can try to start fixing this (hints on where this process happens in
the libs appreciated), but basically I'm wondering, is all of GRASS
like this? If so, I don't have the time to fix what may be a huge job,
but can we at least make the libraries responsible for writing maps
complain (or fail) when a provided map name is too long?

Cheers,

Hamish · July 2, 2010, 7:19am

Hi Joel,

[sorry about yahoo's broken linewrap..]

Joel Pitt wrote:

So I've been getting into all sorts of strife with
hard-coded limits
on filename lengths in GRASS. Not hard-coded in a sense
that they are
enforced (that wouldn't be an issue, because then I'd know
about it!),
but in the sense that there are all these fixed buffer
limits in
modules which don't gracefully handle filenames that get
too long.
Instead it fails with cryptic messages, and since I've
built a large
simulation framework/pipeline around GRASS, it makes
everything else
fall over too.

I've been having trouble with r.random and r.reclass:

http://trac.osgeo.org/grass/ticket/800

I "fixed" those, but now the cell header for the reclassed
map results in:

WARNING: Unable to open raster map

whenever I try to access it. Except, the header in cellhd
looks
exactly the same as another file that's reclassed and works
fine. The
only difference is that the working one reclasses a map
with a short
name.

I can try to start fixing this (hints on where this process
happens in
the libs appreciated), but basically I'm wondering, is all
of GRASS
like this? If so, I don't have the time to fix what may be
a huge job,
but can we at least make the libraries responsible for
writing maps
complain (or fail) when a provided map name is too long?

include/gis.h defines:

/* File/directory name lengths */
#define GNAME_MAX 256
#define GMAPSET_MAX 256

#define GPATH_MAX 4096

but little checks that you haven't exceeded that. We think we've
updated most of the code to use those constants instead of local
ad hoc (smaller) constants, but there will still be many we
missed. If so, the thing to do is to file a bug report and/or
patch. (thanks for ticket #800)

snprintf() stuff is still a bit of an unresolved sore spot but
seems to me not so impossible to fix. maybe a better solution
is in grass7 already, I'm not sure, but at least the ugly write-
to-file hack is removed there AFAIK.

how long are your map+mapset names?

as you found, on top of that, grass 6 titles (cats+hist) are
going to be limited to RECORD_LEN chars. an update of the meta-
data backend remains a major TODO for grass 7.

Hamish

ps- c++ //comments are disfavoured in favour of /* c */ ones.
see the SUBMITTING file.

Glynn_Clements1 · July 2, 2010, 4:00pm

Hamish wrote:

as you found, on top of that, grass 6 titles (cats+hist) are
going to be limited to RECORD_LEN chars. an update of the meta-
data backend remains a major TODO for grass 7.

FWIW, I've re-written this in r42695.

The fields of "struct History" are now dynamically allocated, rather
than fixed-size arrays, so RECORD_LEN and MAXEDLINES no longer exist.

Also, set/get functions are provided, eliminating the need to access
the structure fields directly.

Code which simply truncated strings to RECORD_LEN now stores the
complete string. Code which breaks long strings into multiple lines
has been preserved.

--
Glynn Clements <glynn@gclements.plus.com>

Hamish · July 4, 2010, 5:48am

Hamish wrote:
> as you found, on top of that, grass 6 titles (cats+hist) are
> going to be limited to RECORD_LEN chars. an update of the meta-
> data backend remains a major TODO for grass 7.

Glynn:

FWIW, I've re-written this in r42695.

The fields of "struct History" are now dynamically allocated, rather
than fixed-size arrays, so RECORD_LEN and MAXEDLINES no longer exist.

Also, set/get functions are provided, eliminating the need
to access the structure fields directly.

Code which simply truncated strings to RECORD_LEN now stores the
complete string. Code which breaks long strings into
multiple lines has been preserved.

as always, bloody excellent.

does anyone know/is involved with an OSGeo-wide library/project to deal
with the EU's INSPIRE metadata needs etc? (I'd really rather not attempt
to maintain our own fork of such a thing)
[e.g. something with GeoNetwork http://www.osgeo.org/geonetwork\]

Hamish

Joel_Pitt · July 6, 2010, 4:17am

On Sun, Jul 4, 2010 at 5:48 PM, Hamish <hamish_b@yahoo.com> wrote:

Hamish wrote:
> as you found, on top of that, grass 6 titles (cats+hist) are
> going to be limited to RECORD_LEN chars. an update of the meta-
> data backend remains a major TODO for grass 7.

Glynn:

FWIW, I've re-written this in r42695.

The fields of "struct History" are now dynamically allocated, rather
than fixed-size arrays, so RECORD_LEN and MAXEDLINES no longer exist.

Also, set/get functions are provided, eliminating the need
to access the structure fields directly.

Code which simply truncated strings to RECORD_LEN now stores the
complete string. Code which breaks long strings into
multiple lines has been preserved.

as always, bloody excellent.

does anyone know/is involved with an OSGeo-wide library/project to deal
with the EU's INSPIRE metadata needs etc? (I'd really rather not attempt
to maintain our own fork of such a thing)
[e.g. something with GeoNetwork http://www.osgeo.org/geonetwork\]

Thanks Hamish for the explanation and the reference to where in the
code the limits are (were) kept.

Thanks Glynn for the move to to dynamically allocated history fields.

In answer to the length of the filenames I was using... they were too
long generally and I realised that I could avoid them. I was storing
model outputs which included parameters, time step, model names etc to
differentiate. Now I use separate mapsets for each specific
combination of parameters and just use a subdirectory in the mapset
for storing model metadata/analysis.

edit: I just looked up some model logs where I was having the issue
and the files causing the problems were 155 char long in a mapset 24
chars long.

J