[GRASS-dev] [GRASS GIS] #3832: Add environment variable to create all temp-files at a user-defined location (outside the mapsets)

#3832: Add environment variable to create all temp-files at a user-defined
location (outside the mapsets)
-----------------------------------+-------------------------
Reporter: sbl | Owner: grass-dev@…
     Type: enhancement | Status: new
Priority: normal | Milestone: 7.8.0
Component: Default | Version: svn-trunk
Keywords: environment variables | CPU: Unspecified
Platform: All |
-----------------------------------+-------------------------
Some Network Attached Storage (NAS) filers with snapshot backup on volume
level do not allow to exclude specific file types or directories to be
excluded from backup.
If GRASS data DB is stored on such a storage system, temporary files can
grow the snapshot file significantly until a point where available space
is used up by backup files (how fast that happens depends on the snapshot
frequency of course, and the time they are preserved).

Currently, it is not possible to redirect all temporary data to other
storage locations. Would be great to have an environment variable that
allows to specify a path (directory) where all temporary data will be
stored.

Might be necessary to make a subdirectory per mapset within that
directory... (Maybe an 8.0 thing?)

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3832&gt;
GRASS GIS <https://grass.osgeo.org>

#3832: Add environment variable to create all temp-files at a user-defined
location (outside the mapsets)
--------------------------+-----------------------------------
  Reporter: sbl | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Default | Version: svn-trunk
Resolution: | Keywords: environment variables
       CPU: Unspecified | Platform: All
--------------------------+-----------------------------------

Comment (by neteler):

As a workaround you could simply replace the location/mapset/.tmp/
with a link to a directory on SSD.

However, from the planning document:

https://trac.osgeo.org/grass/wiki/Grass8Planning#Rasterlibrary

  * "implement GRASS_RASTER_TMPDIR_MAPSET like it exists for vector data
(GRASS_VECTOR_TMPDIR_MAPSET), i.e. change all .tmp/ to variable in source
code in init/grass.py, gis/open.c, gis/file_name.c, raster library"

Maybe to be worked out in more detail but for sure very important (we
frequently face the same issue).

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3832#comment:1&gt;
GRASS GIS <https://grass.osgeo.org>

Hi Markus,

And thanks for the info/feedback.
Although the redirection of the .tmp directory (in future through an environment variable hopefully) already helps quite a bit and is a rather simple remedy (from the user perspective), intermediate maps can still cause similar issues.

Inspired by g.cloud, I tend to setup a local GRASS DB on SSD, where I link / copy mapsets from network storage as needed for the process. That way, (script based) processing does not affect snapshot backup at all. So, my follow up question is if a Python function to create a local grass DB from a central GRASS DB on network storage might be useful.

I am thinking along the following lines:

def setup_local_db(local_db_path, network_db_path, link_locations_mapsets, copy_locations_mapsets):
    """creates a local GRASS DB by linking or copying a user defined list of mapsets from a GRASS DB on network storage
        :param local_db_path: path where the local GRASS DB should be created
        :type local_db_path: str
        :param network_db_path: path to the GRASS DB on network storage from which mapsest should be linked / copied
        :type network_db_path: str
        :param link_locations_mapsets: tuple with combinations of (location, mapset) to be linked to local GRASS DB
        :type link_locations_mapsets: tuple
        :param copy_locations_mapsets: tuple with combinations of (location, mapset) to be copied to local GRASS DB
        :type copy_locations_mapsets: tuple
    """

Would that be suitable for the GRASS code base? Such a function seems little helpful to run from within an existing GRASS session, so an addon does not seem to make very much sense either, right? Maybe if Panos work on making grass modules importable without existing session is successful, one could setup local db and then start a session. But maybe this would be more suitable for grass-session?

Any comments recommendations, suggestions are most welcome...

Cheers,
Stefan

-----Original Message-----
From: GRASS GIS <trac@osgeo.org>
Sent: torsdag 2. mai 2019 15:21
Cc: grass-dev@lists.osgeo.org
Subject: Re: [GRASS GIS] #3832: Add environment variable to create all temp-files at a user-defined location (outside the mapsets)

#3832: Add environment variable to create all temp-files at a user-defined location (outside the mapsets)
--------------------------+-----------------------------------
  Reporter: sbl | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Default | Version: svn-trunk
Resolution: | Keywords: environment variables
       CPU: Unspecified | Platform: All
--------------------------+-----------------------------------

Comment (by neteler):

As a workaround you could simply replace the location/mapset/.tmp/ with a link to a directory on SSD.

However, from the planning document:

https://trac.osgeo.org/grass/wiki/Grass8Planning#Rasterlibrary

  * "implement GRASS_RASTER_TMPDIR_MAPSET like it exists for vector data (GRASS_VECTOR_TMPDIR_MAPSET), i.e. change all .tmp/ to variable in source code in init/grass.py, gis/open.c, gis/file_name.c, raster library"

Maybe to be worked out in more detail but for sure very important (we frequently face the same issue).

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3832#comment:1&gt;
GRASS GIS <https://grass.osgeo.org>

#3832: Add environment variable to create all temp-files at a user-defined
location (outside the mapsets)
--------------------------+-----------------------------------
  Reporter: sbl | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Default | Version: svn-trunk
Resolution: | Keywords: environment variables
       CPU: Unspecified | Platform: All
--------------------------+-----------------------------------

Comment (by sbl):

Consider also:
https://trac.osgeo.org/grass/ticket/3857#comment:4

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3832#comment:2&gt;
GRASS GIS <https://grass.osgeo.org>

#3832: Add environment variable to create all temp-files at a user-defined
location (outside the mapsets)
--------------------------+-----------------------------------
  Reporter: sbl | Owner: grass-dev@…
      Type: enhancement | Status: new
  Priority: normal | Milestone: 7.8.0
Component: Default | Version: svn-trunk
Resolution: | Keywords: environment variables
       CPU: Unspecified | Platform: All
--------------------------+-----------------------------------

Comment (by martinl):

Such variable is currently implemented only for vector library
(GRASS_VECTOR_TMPDIR_MAPSET), see also related discussion,
https://lists.osgeo.org/pipermail/grass-dev/2015-June/075220.html

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3832#comment:3&gt;
GRASS GIS <https://grass.osgeo.org>