[GRASS-dev] TGRASS: mapset management question

neteler · September 12, 2015, 7:28pm

Hi again,

I am trying to understand if there is a way to use a TGIS DB from
another mapset in the same location.
(yes I am aware of the original PERMANENT-takes-it-all implementation
but that was against the file handling architecture of the rest of
GRASS GIS)

Rationale: I have a mapset with 20k maps of actual MODIS LST data,
each map comes with 415 million pixels (5.2TB at time).

From that I derive daily, weekly, monthly, seasonal averages, GDD,

Winkler, BEDD, etc. etc. Each of these averages or index (covering 15
years) goes into its own mapset for practical reasons (mapset size,
amount of maps). Altogether at time more than 18TB.

So it would be ideal to read the master TGIS DB in the next new mapset.

I tried to solve that by a link but this failed. Is there a way to do
that? Essentially it would be e.g.

t.info my_tgis@the_other_mapset

thanks
Markus

neteler · September 14, 2015, 8:53pm

Hi Soeren,

thanks for fixes r66205 + r66206, now I get at least "some" feedback

GRASS 7.1.svn (eu_laea):~ > t.list
WARNING: You don't have permission to access the mapset
         <modis_lst_reconstructed>
WARNING: You don't have permission to access the mapset
         <modis_lst_reconstructed>
----------------------------------------------
WARNING: You don't have permission to access the mapset
         <modis_lst_reconstructed>
WARNING: You don't have permission to access the mapset
         <modis_lst_reconstructed>
WARNING: You don't have permission to access the mapset
         <modis_lst_reconstructed>
WARNING: You don't have permission to access the mapset
         <modis_lst_reconstructed>
Space time raster datasets with absolute time available in mapset
<modis_lst_reconstructed>:
modis_lst_reconstructed@modis_lst_reconstructed
WARNING: You don't have permission to access the mapset
         <modis_lst_reconstructed>
WARNING: You don't have permission to access the mapset
         <modis_lst_reconstructed>
WARNING: You don't have permission to access the mapset
         <modis_lst_reconstructed>
WARNING: You don't have permission to access the mapset
         <modis_lst_reconstructed>

Yet the stdrs is reported, great.

Is the warning meant as write access limitation?

You pointed me offlist to the env var GRASS_SKIP_MAPSET_OWNER_CHECK mentioned in
https://grass.osgeo.org/grass71/manuals/variables.html
which indeed changes this

GRASS 7.1.svn (eu_laea):~ > export GRASS_SKIP_MAPSET_OWNER_CHECK=1
GRASS 7.1.svn (eu_laea):~ > t.list
----------------------------------------------
Space time raster datasets with absolute time available in mapset
<modis_lst_reconstructed>:
modis_lst_reconstructed@modis_lst_reconstructed

I wonder why I need to set GRASS_SKIP_MAPSET_OWNER_CHECK=1 when I just
want to read something from the other mapset.

I see that is is (only) used in
lib/gis/mapset_msc.c
in the check_owner() function which is called by
G_mapset_permissions() which is called in
lib/python/temporal/c_libraries_interface.py

If the point of using that env var is to simply suppress the warning
in the upper case, then maybe G_suppress_warnings(1) is better?
Something like

Index: lib/python/temporal/c_libraries_interface.py

--- lib/python/temporal/c_libraries_interface.py (revision 66229)
+++ lib/python/temporal/c_libraries_interface.py (working copy)
@@ -294,7 +294,9 @@
char_list = ""
mapset = mapsets[count]

+ libgis.G_suppress_warnings(1)
permission = libgis.G_mapset_permissions(mapset)
+ libgis.G_suppress_warnings(0)
in_search_path = libgis.G_is_mapset_in_search_path(mapset)

I see it used in various raster modules.
What do you think?

In any case I can report that it works, very good!

GRASS 7.1.svn (eu_laea):~ > t.rast.list
input=modis_lst_reconstructed@modis_lst_reconstructed columns=name
method=comma separator="newline" where="start_time >
'2014-01-03 17:00:00' and start_time < '2014-01-05 07:00:00'"
WARNING: You don't have permission to access the mapset
         <modis_lst_reconstructed>
WARNING: You don't have permission to access the mapset
         <modis_lst_reconstructed>
WARNING: You don't have permission to access the mapset
         <modis_lst_reconstructed>
WARNING: You don't have permission to access the mapset
         <modis_lst_reconstructed>
MOD11A1.A2014003.LST_Night_1km.reconstruct@modis_lst_reconstructed,MYD11A1.A2014004.LST_Night_1km.reconstruct@modis_lst_reconstructed,MOD11A1.A2014004.LST_Day_1km.reconstruct@modis_lst_reconstructed,MYD11A1.A2014004.LST_Day_1km.reconstruct@modis_lst_reconstructed,MOD11A1.A2014004.LST_Night_1km.reconstruct@modis_lst_reconstructed,MYD11A1.A2014005.LST_Night_1km.reconstruct@modis_lst_reconstructed

So, the remaining issue is just the WARNING.
Working with TGIS data stored in another DB now works, this is really
great and a big help!

thanks again,
Markus

wenzeslaus · September 14, 2015, 9:51pm

On Mon, Sep 14, 2015 at 4:53 PM, Markus Neteler <neteler@osgeo.org> wrote:

If the point of using that env var is to simply suppress the warning
in the upper case, then maybe G_suppress_warnings(1) is better?
Something like

Index: lib/python/temporal/c_libraries_interface.py

— lib/python/temporal/c_libraries_interface.py (revision 66229)
+++ lib/python/temporal/c_libraries_interface.py (working copy)
@@ -294,7 +294,9 @@
char_list = “”
mapset = mapsets[count]

libgis.G_suppress_warnings(1)
permission = libgis.G_mapset_permissions(mapset)

libgis.G_suppress_warnings(0)
in_search_path = libgis.G_is_mapset_in_search_path(mapset)

I see it used in various raster modules.
What do you think?

Well, if it is in multiple raster modules then it is a conceptual problem. Either the modules are doing something wrong or the right API is missing for that case.

Vaclav

Soeren_Gebbert2 · September 15, 2015, 7:06am

Hi

2015-09-14 23:51 GMT+02:00 Vaclav Petras <wenzeslaus@gmail.com>:

On Mon, Sep 14, 2015 at 4:53 PM, Markus Neteler <neteler@osgeo.org> wrote:

If the point of using that env var is to simply suppress the warning
in the upper case, then maybe G_suppress_warnings(1) is better?
Something like

Index: lib/python/temporal/c_libraries_interface.py

--- lib/python/temporal/c_libraries_interface.py (revision 66229)
+++ lib/python/temporal/c_libraries_interface.py (working copy)
@@ -294,7 +294,9 @@
             char_list = ""
             mapset = mapsets[count]

+ libgis.G_suppress_warnings(1)
             permission = libgis.G_mapset_permissions(mapset)
+ libgis.G_suppress_warnings(0)
             in_search_path = libgis.G_is_mapset_in_search_path(mapset)

I see it used in various raster modules.
What do you think?

Well, if it is in multiple raster modules then it is a conceptual problem.
Either the modules are doing something wrong or the right API is missing for
that case.

I am not sure about this, but i have the feeling that the
mapset/permission API needs some love. It was not obvious for me, that
i need to call
G_get_mapset_name(0) to initialize an internal static list so that
G_is_mapset_in_search_path() will work. That is bad design and not
well documented.
Regarding G_mapset_permissions(), it only checks the owner but not the
permission for groups and other, hence it is only used to determine if
the mapset exists
as in g.list.

However, the warning about mapset access is generated in
"lib/temporal/lib/connect.c" which checks the output of
G_mapset_permissions2. I will remove this check
since only the owner is checked, not access permissions.

Ciao
Sören

Vaclav

_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

wenzeslaus · September 15, 2015, 5:13pm

On Tue, Sep 15, 2015 at 3:06 AM, Sören Gebbert <soerengebbert@googlemail.com> wrote:

However, the warning about mapset access is generated in
“lib/temporal/lib/connect.c” which checks the output of
G_mapset_permissions2. I will remove this check
since only the owner is checked, not access permissions.

Be aware of the two different access permissions concepts in GRASS.

file permissions as set by the system
user permissions for Mapsets managed by GRASS

The fist is pure system thing. GRASS probably should check that (I don’t know if this actually happens).

The second is partially file ownership as set in the system and partially GRASS Mapset permissions for users. User can only open Mapset which belongs to him or her, unless the owner explicitly allows this in GRASS.

If this is good or bad, I don’t know, but it is the current situation.

neteler · September 17, 2015, 11:53am

Hi,

On Mon, Sep 14, 2015 at 10:53 PM, Markus Neteler <neteler@osgeo.org> wrote:

Hi Soeren,

thanks for fixes r66205 + r66206, now I get at least "some" feedback

I have tried to backport this to the relbranch in
https://trac.osgeo.org/grass/changeset/66250
but still no luck. What am I missing to see other mapsets also there
for the t.* modules?

thanks
Markus

neteler · September 17, 2015, 9:19pm

On Thu, Sep 17, 2015 at 1:53 PM, Markus Neteler <neteler@osgeo.org> wrote:

Hi,

On Mon, Sep 14, 2015 at 10:53 PM, Markus Neteler <neteler@osgeo.org> wrote:

Hi Soeren,

thanks for fixes r66205 + r66206, now I get at least "some" feedback

I have tried to backport this to the relbranch in
https://trac.osgeo.org/grass/changeset/66250
but still no luck.

Indeed yes:

GRASS 7.0.2svn (eu_laea):~ > t.rast.list
input=modis_lst_reconstructed@modis_lst_reconstructed columns=name
method=comma separator="newline" where="start_time >
'2014-01-03 17:00:00' and start_time < '2014-01-05 07:00:00'"
WARNING: You don't have permission to access the mapset
         <modis_lst_reconstructed>
WARNING: You don't have permission to access the mapset
         <modis_lst_reconstructed>
WARNING: You don't have permission to access the mapset
         <modis_lst_reconstructed>
WARNING: You don't have permission to access the mapset
         <modis_lst_reconstructed>
MOD11A1.A2014003.LST_Night_1km.reconstruct@modis_lst_reconstructed,MYD11A1.A2014004.LST_Night_1km.reconstruct@modis_lst_reconstructed,MOD11A1.A2014004.LST_Day_1km.reconstruct@modis_lst_reconstructed,MYD11A1.A2014004.LST_Day_1km.reconstruct@modis_lst_reconstructed,MOD11A1.A2014004.LST_Night_1km.reconstruct@modis_lst_reconstructed,MYD11A1.A2014005.LST_Night_1km.reconstruct@modis_lst_reconstructed

So, multiple mapset support backported.

Thanks again to Soeren to figure it out.

best
Markus

PS: Now we only need to shut up the warnings there

neteler · September 21, 2015, 9:06pm

On Thu, Sep 17, 2015 at 11:19 PM, Markus Neteler <neteler@osgeo.org> wrote:

On Thu, Sep 17, 2015 at 1:53 PM, Markus Neteler <neteler@osgeo.org> wrote:

...

So, multiple mapset support backported.

Thanks again to Soeren to figure it out.

...

PS: Now we only need to shut up the warnings there

Found it:
lib/temporal/lib/connect.c

static char *get_mapset_connection_name(const char *mapset, int contype)
{
    const char *val = NULL;
    char *ret_val = NULL;;
    const char *gisdbase = G_getenv_nofatal("GISDBASE");
    const char *location = G_getenv_nofatal("LOCATION_NAME");
    int ret;

    G_debug(1,"Checking mapset <%s>", mapset);
    ret = G_mapset_permissions2(gisdbase, location, mapset);
    switch (ret) {
    case 0: /* Check if the mapset exists and user is owner */
        //G_warning(_("You don't have permission to access the mapset <%s>"),
        // mapset);
        break;
    case -1:
        G_warning(_("Mapset <%s> does not exist."),
                      mapset);
        break;
    default:
        break;
    }
...

The problem is that in case of e.g. t.rast.list the
G_mapset_permissions2() test is too "agressive" since it checks if I
am the *owner* of another mapset (I am not in my case) while I just
want to read data from there.

Since we have 160+ mapsets in our EU LAEA location generated and owned
by a series of owners, I get spammed with messages of above case 0.
In the end the result of e.g. t.rast.list is almost invisible.

Is there a way to add an extra condition within the case 0. Or do we
need a new G_mapset_permissions3() being less picky about ownership?

Markus

wenzeslaus · September 21, 2015, 11:57pm

On Mon, Sep 21, 2015 at 5:06 PM, Markus Neteler <neteler@osgeo.org> wrote:

Is there a way to add an extra condition within the case 0. Or do we
need a new G_mapset_permissions3() being less picky about ownership?

I'm not completely following the discussion but if there is a need to test
Mapset permissions for read-only in GRASS way (not just system
permissions), there should be a function to do that.

neteler · September 22, 2015, 1:35pm

On Tue, Sep 22, 2015 at 1:57 AM, Vaclav Petras <wenzeslaus@gmail.com> wrote:

On Mon, Sep 21, 2015 at 5:06 PM, Markus Neteler <neteler@osgeo.org> wrote:

Is there a way to add an extra condition within the case 0. Or do we
need a new G_mapset_permissions3() being less picky about ownership?

I'm not completely following the discussion but if there is a need to test
Mapset permissions for read-only in GRASS way (not just system permissions),
there should be a function to do that.

I'm not aware of any. But we need that in my opinion.

Markus