[GRASS-dev] Re: [GRASS-user] Cannot use grass on vfat partition

GRASS GRASS developer
1

Agustin Lobo wrote:

I'm using grass 6.2.3 on ubuntu 7.1
I've set up a fat32 partition so that I can share data with windows.
The line in my fstab is:
/dev/sda4 /media/mifat32 vfat auto,rw,gid=1000,uid=1000,iocharset=utf8,umask=0000 0 0

I mount with no problems, files belong to user alobo group alobo, I
can write,delete and make directories.

When I try to create a LOCATION and a mapset, I get errors. For
example, cannot do it with
the graphic interface, if I select Create mapset, I get:

couldn't change working directory to "##ERROR##": no such file or directory
couldn't change working directory to "##ERROR##": no such file or directory
    while executing
"cd $location"
    invoked from within
".frame0.frameNMS.third.button invoke"
    ("uplevel" body line 1)
    invoked from within
"uplevel #0 [list $w invoke]"
    (procedure "tk::ButtonUp" line 22)
    invoked from within
"tk::ButtonUp .frame0.frameNMS.third.button"
    (command bound to event)

Just a guess, but is the PERMANENT directory in upper or lower case?

If Linux has decided to convert it to lower case, the CheckLocation
procedure in gis_set.tcl won't find it.

FWIW (CC to grass-dev), checking for PERMANENT like this:

        foreach filename [lsort [glob -nocomplain *]] {
            if {[string compare $filename "PERMANENT"] == 0} {
    ...
            }
        }

is bogus. It should be:

  if {[file isdirectory "PERMANENT"]} {
    ...
  }

The latter will correctly handle a filesystem which isn't
case-sensitive.

It should also handle the case where you don't have read permission on
the location directory. Enumeration requires read permission, while
lookup only requires execute permission. You shouldn't actually need
read permission for anything other than "g.mapsets -l".

I can do it with the console (Projection Values), but then, at the end,
the graphic interface quits and I get:

GRASS 6.2.3 (delmedir):/media/mifat32 > ERROR: default region is not set

This might be related; if it's looking for PERMANENT/DEFAULT_WIND
using enumeration and comparison (as above), it will fail if the case
is wrong.

--
Glynn Clements <glynn@gclements.plus.com>

2

Agustin Lobo wrote:

Also, if within the grass text session I do:
  g.mapset /media/mifat32/test/testmapset
I get:
ERROR: The mapset does not exist.

That's correct. The mapset= argument should be the name of a mapset
(e.g. "testmapset"), not a full path.

Just found the difference between locations on
the vfat and on the ext3 partition:
mapsets on the vfat partition get
files MYNAME WIND as myname and wind.

That explains it.

Just renaming these files makes everything work.

What I do not understand is why grass
writes thes files in lower case, it does not
with LOCATION for example, and even less
why grass has a different behaviour
for different partition formats

It isn't GRASS that's doing this, but Linux.

The reason why it only does it to those files and not to PERMANENT and
DEFAULT_WIND is the length of the name.

If you look at the mount(8) manual page, in the section entitled
"Mount options for vfat", it says:

       shortname=[lower|win95|winnt|mixed]

              Defines the behaviour for creation and display of filenames
              which fit into 8.3 characters. If a long name for a file exists,
              it will always be preferred display. There are four modes:

              lower Force the short name to lower case upon display; store a
                     long name when the short name is not all upper case.

              win95 Force the short name to upper case upon display; store a
                     long name when the short name is not all upper case.

              winnt Display the shortname as is; store a long name when the
                     short name is not all lower case or all upper case.

              mixed Display the short name as is; store a long name when the
                     short name is not all upper case.

       The default is "lower".

With the default setting of "lower", MYNAME and WIND fit into 8.3
format (up to 8 characters, with an optional extension of up to 3
characters), and are all upper-case, so the VFAT filesystem driver
only creates a normal "DOS" filename, and not a VFAT long filename
(LFN). When the directory is enumerated, the short filenames are
returned as lower case (the on-disk format uses upper case).

If you add shortname=win95 to the mount options, the issue should go
away. If the name is all upper case, only a short name will be stored,
but the short name will be returned in upper case. If the name isn't
all upper case, a long name will be stored, and the long name will
always be returned as-is; the short name will be ignored.

Using shortname=winnt would also work, but that results in a mix of
upper- and lower-case short names on disk, which is less compatible
(MS-DOS only ever used upper case for the on-disk format).

BTW, I have no idea why the default is "lower".

Even so, this still indicates a bug in GRASS; i.e. something is
enumerating the directory and expecting to see a file named "WIND",
rather than asking the OS whether "WIND" exists. This is wrong, for
the reasons which I outlined previously.

--
Glynn Clements <glynn@gclements.plus.com>

3

Agustin Lobo wrote:

Thanks, solved by adding shortname=win95 to the mount options in /etc/fstab

Should a bug report be filled? by me?
Should I write a note in the wiki?

A note in the Wiki would be good, if you can figure out where to put
it.

A bug report is problematic because it isn't clear where the bug lies.
The handling of PERMANENT is a bug, but won't normally get triggered
in practice. I can't see where the incorrect test for MYNAME or WIND
lies.

--
Glynn Clements <glynn@gclements.plus.com>

4

Hi all,

I changed that ugly PERMANENT check in gis_set.tcl file, but IMHO it
will not help in this case, as tcl (gis_set.tcl), shell (init.sh) and
C modules (g.region and firends) still will work in case-sensitive way
in case-sensitive systems (Linux and other Unix-like systems). IMHO
this should be just added to list of known issues.
Correct me, if I'm wrong.

Maris.

2008/5/2, Glynn Clements <glynn@gclements.plus.com>:

Agustin Lobo wrote:

> Thanks, solved by adding shortname=win95 to the mount options in
/etc/fstab
>
> Should a bug report be filled? by me?
> Should I write a note in the wiki?

A note in the Wiki would be good, if you can figure out where to put
it.

A bug report is problematic because it isn't clear where the bug lies.
The handling of PERMANENT is a bug, but won't normally get triggered
in practice. I can't see where the incorrect test for MYNAME or WIND
lies.

--
Glynn Clements <glynn@gclements.plus.com>
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

5

Maris Nartiss wrote:

I changed that ugly PERMANENT check in gis_set.tcl file, but IMHO it
will not help in this case, as tcl (gis_set.tcl), shell (init.sh) and
C modules (g.region and firends) still will work in case-sensitive way
in case-sensitive systems (Linux and other Unix-like systems). IMHO
this should be just added to list of known issues.
Correct me, if I'm wrong.

If a program accesses (open(), stat(), access(), etc) a file on a
case-insensitive filesystem, the access will work regardless of case.

If you create a file named "WIND" on a FAT (or SMB) filesystem, and
the filename gets converted to "wind", calling e.g. open("WIND",...)
will still open the file. The filesystem driver knows whether the
filesystem is case-sensitive or not.

The problem comes when the code tries to do the lookup itself, by
enumerating the directory (opendir(), readdir(), closedir()) then
matching the filename with e.g. strcmp(). The program cannot easily
tell whether the directory resides on a case-sensitive filesystem. You
can't just use "#ifdef __MINGW32__", as Linux can access FAT
partitions and SMB shares, while Windows systems can access Samba
servers which export case-sensitive filesystems (also, NTFS allows
case-sensitive directories, but they aren't often used, as a lot of
Windows software doesn't handle this correctly).

Apart from the issue of case-sensitivity, there's also the fact that
enumeration requires more privilege than lookup[1], and it may also
have implications regarding Unicode normalisation[2].

[1] I doubt that this will be an issue for GRASS databases, but on
shared servers it's quite common for "system" directories to be mode
711 (drwx--x--x), so processes owned by normal users can access known
system files but cannot enumerate the directories.

[2] On OSX, accented characters (in filenames) are stored as a
combination of a non-accented character and a composing accent
character. The higher-level library functions (I don't know about
fopen() etc) automatically normalise filenames, so the string returned
from readdir() won't necessarily match the string passed to the
function which created the file.

--
Glynn Clements <glynn@gclements.plus.com>

6

Glynn,

Unfortunately, this is not the case. Directory
drwxrwxrwx /media/mifat32/test/PERMANENT
is in upper case, with its
DEFAULT_WIND myname wind files
all of them with -rwxrwxrwx permissions.

If I start
grass -text
on /media/mifat32
I enter a normal grass session (although g.mapset does not work!)

But if I do
grass -gui
I get the gui with
GIS data directory: /media/mifat32
Project Location: test
Mapset: testmapset

and when I select Enter GRASS, I get an error:
Warning Invalid Mapset
Warning <0> is not a valid mapset.

Also, if within the grass text session I do:
  g.mapset /media/mifat32/test/testmapset
I get:
ERROR: The mapset does not exist.

even if I do it from within the
/media/mifat32/test/testmapset directory!

This is so confusing...

Agus

Glynn Clements wrote:

Agustin Lobo wrote:

I'm using grass 6.2.3 on ubuntu 7.1
I've set up a fat32 partition so that I can share data with windows.
The line in my fstab is:
/dev/sda4 /media/mifat32 vfat auto,rw,gid=1000,uid=1000,iocharset=utf8,umask=0000 0 0

I mount with no problems, files belong to user alobo group alobo, I
can write,delete and make directories.

When I try to create a LOCATION and a mapset, I get errors. For
example, cannot do it with
the graphic interface, if I select Create mapset, I get:

couldn't change working directory to "##ERROR##": no such file or directory
    while executing
"cd $location"
    invoked from within
".frame0.frameNMS.third.button invoke"
    ("uplevel" body line 1)
    invoked from within
"uplevel #0 [list $w invoke]"
    (procedure "tk::ButtonUp" line 22)
    invoked from within
"tk::ButtonUp .frame0.frameNMS.third.button"
    (command bound to event)

Just a guess, but is the PERMANENT directory in upper or lower case?

If Linux has decided to convert it to lower case, the CheckLocation
procedure in gis_set.tcl won't find it.

FWIW (CC to grass-dev), checking for PERMANENT like this:

        foreach filename [lsort [glob -nocomplain *]] {
            if {[string compare $filename "PERMANENT"] == 0} {
    ...
            }
        }

is bogus. It should be:

  if {[file isdirectory "PERMANENT"]} {
    ...
  }

The latter will correctly handle a filesystem which isn't
case-sensitive.

It should also handle the case where you don't have read permission on
the location directory. Enumeration requires read permission, while
lookup only requires execute permission. You shouldn't actually need
read permission for anything other than "g.mapsets -l".

I can do it with the console (Projection Values), but then, at the end,
the graphic interface quits and I get:

GRASS 6.2.3 (delmedir):/media/mifat32 > ERROR: default region is not set

This might be related; if it's looking for PERMANENT/DEFAULT_WIND
using enumeration and comparison (as above), it will fail if the case
is wrong.

--
Dr. Agustin Lobo
Institut de Ciencies de la Terra "Jaume Almera" (CSIC)
LLuis Sole Sabaris s/n
08028 Barcelona
Spain
Tel. 34 934095410
Fax. 34 934110012
email: Agustin.Lobo@ija.csic.es
http://www.ija.csic.es/gt/obster

7

Just found the difference between locations on
the vfat and on the ext3 partition:
mapsets on the vfat partition get
files MYNAME WIND as myname and wind.

Just renaming these files makes everything work.

What I do not understand is why grass
writes thes files in lower case, it does not
with LOCATION for example, and even less
why grass has a different behaviour
for different partition formats

Note that in both cases I'm using linux (Ubuntu 8.04)
and grass6.2.3

Thanks

Agus

Agustin Lobo wrote:

Glynn,

Unfortunately, this is not the case. Directory
drwxrwxrwx /media/mifat32/test/PERMANENT
is in upper case, with its
DEFAULT_WIND myname wind files
all of them with -rwxrwxrwx permissions.

If I start
grass -text
on /media/mifat32
I enter a normal grass session (although g.mapset does not work!)

But if I do
grass -gui
I get the gui with
GIS data directory: /media/mifat32
Project Location: test
Mapset: testmapset

and when I select Enter GRASS, I get an error:
Warning Invalid Mapset
Warning <0> is not a valid mapset.

Also, if within the grass text session I do:
g.mapset /media/mifat32/test/testmapset
I get:
ERROR: The mapset does not exist.

even if I do it from within the
/media/mifat32/test/testmapset directory!

This is so confusing...

Agus

Glynn Clements wrote:

Agustin Lobo wrote:

I'm using grass 6.2.3 on ubuntu 7.1
I've set up a fat32 partition so that I can share data with windows.
The line in my fstab is:
/dev/sda4 /media/mifat32 vfat auto,rw,gid=1000,uid=1000,iocharset=utf8,umask=0000 0 0

I mount with no problems, files belong to user alobo group alobo, I
can write,delete and make directories.

When I try to create a LOCATION and a mapset, I get errors. For
example, cannot do it with
the graphic interface, if I select Create mapset, I get:

couldn't change working directory to "##ERROR##": no such file or directory
couldn't change working directory to "##ERROR##": no such file or directory
    while executing
"cd $location"
    invoked from within
".frame0.frameNMS.third.button invoke"
    ("uplevel" body line 1)
    invoked from within
"uplevel #0 [list $w invoke]"
    (procedure "tk::ButtonUp" line 22)
    invoked from within
"tk::ButtonUp .frame0.frameNMS.third.button"
    (command bound to event)

Just a guess, but is the PERMANENT directory in upper or lower case?

If Linux has decided to convert it to lower case, the CheckLocation
procedure in gis_set.tcl won't find it.

FWIW (CC to grass-dev), checking for PERMANENT like this:

        foreach filename [lsort [glob -nocomplain *]] {
            if {[string compare $filename "PERMANENT"] == 0} {
        ...
            }
        }

is bogus. It should be:

    if {[file isdirectory "PERMANENT"]} {
        ...
    }

The latter will correctly handle a filesystem which isn't
case-sensitive.

It should also handle the case where you don't have read permission on
the location directory. Enumeration requires read permission, while
lookup only requires execute permission. You shouldn't actually need
read permission for anything other than "g.mapsets -l".

I can do it with the console (Projection Values), but then, at the end,
the graphic interface quits and I get:

GRASS 6.2.3 (delmedir):/media/mifat32 > ERROR: default region is not set

This might be related; if it's looking for PERMANENT/DEFAULT_WIND
using enumeration and comparison (as above), it will fail if the case
is wrong.

--
Dr. Agustin Lobo
Institut de Ciencies de la Terra "Jaume Almera" (CSIC)
LLuis Sole Sabaris s/n
08028 Barcelona
Spain
Tel. 34 934095410
Fax. 34 934110012
email: Agustin.Lobo@ija.csic.es
http://www.ija.csic.es/gt/obster

8

Thanks, solved by adding shortname=win95 to the mount options in /etc/fstab

Should a bug report be filled? by me?
Should I write a note in the wiki?

Agus

Glynn Clements wrote:

Agustin Lobo wrote:

Also, if within the grass text session I do:
  g.mapset /media/mifat32/test/testmapset
I get:
ERROR: The mapset does not exist.

That's correct. The mapset= argument should be the name of a mapset
(e.g. "testmapset"), not a full path.

Just found the difference between locations on
the vfat and on the ext3 partition:
mapsets on the vfat partition get
files MYNAME WIND as myname and wind.

That explains it.

Just renaming these files makes everything work.

What I do not understand is why grass
writes thes files in lower case, it does not
with LOCATION for example, and even less
why grass has a different behaviour
for different partition formats

It isn't GRASS that's doing this, but Linux.

The reason why it only does it to those files and not to PERMANENT and
DEFAULT_WIND is the length of the name.

If you look at the mount(8) manual page, in the section entitled
"Mount options for vfat", it says:

       shortname=[lower|win95|winnt|mixed]

              Defines the behaviour for creation and display of filenames
              which fit into 8.3 characters. If a long name for a file exists,
              it will always be preferred display. There are four modes:

              lower Force the short name to lower case upon display; store a
                     long name when the short name is not all upper case.

              win95 Force the short name to upper case upon display; store a
                     long name when the short name is not all upper case.

              winnt Display the shortname as is; store a long name when the
                     short name is not all lower case or all upper case.

              mixed Display the short name as is; store a long name when the
                     short name is not all upper case.

       The default is "lower".

With the default setting of "lower", MYNAME and WIND fit into 8.3
format (up to 8 characters, with an optional extension of up to 3
characters), and are all upper-case, so the VFAT filesystem driver
only creates a normal "DOS" filename, and not a VFAT long filename
(LFN). When the directory is enumerated, the short filenames are
returned as lower case (the on-disk format uses upper case).

If you add shortname=win95 to the mount options, the issue should go
away. If the name is all upper case, only a short name will be stored,
but the short name will be returned in upper case. If the name isn't
all upper case, a long name will be stored, and the long name will
always be returned as-is; the short name will be ignored.

Using shortname=winnt would also work, but that results in a mix of
upper- and lower-case short names on disk, which is less compatible
(MS-DOS only ever used upper case for the on-disk format).

BTW, I have no idea why the default is "lower".

Even so, this still indicates a bug in GRASS; i.e. something is
enumerating the directory and expecting to see a file named "WIND",
rather than asking the OS whether "WIND" exists. This is wrong, for
the reasons which I outlined previously.