[GRASS-dev] [GRASS GIS] #520: r.terraflow/iostream does not respect STREAM_DIR option

GRASS GRASS developer
21

#520: r.terraflow/iostream does not respect STREAM_DIR option
---------------------+------------------------------------------------------
  Reporter: mmetz | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: Raster | Version: svn-develbranch6
Resolution: | Keywords: r.terraflow, tmp
  Platform: All | Cpu: All
---------------------+------------------------------------------------------
Comment (by mmetz):

Replying to [comment:19 hamish]:
> Replying to [comment:18 mmetz]:
>
> > The default answer of /var/tmp or another system TEMP folder
> > is IMHO not a good idea because of lack of space. Why not
> > leaving it blank, if no answer is given, use the GRASS temp
> > directory? That directory should always exist and I guess there
> > is most of the time enough space.
>
> this has been discussed before, check the archives. ISTR that the
Terraflow team (Andrew? Laura?) explained why they used that a long time
ago (I now forget exact reason),

Picking a directory whose contents are deleted automatically if I remember
right, which would be the case for $MAPSET/.tmp/$MACHINE.

Whatever directory is used as default in r.terraflow, it should at least
exist on all supported platforms.

> Note that these temp files may be >2gig

I get regularly >20GB, having to work on somewhat larger datasets.
Thinking about Doug's datasets, it would go way above 100GB.

> (use the memory= option to lower those needs), and they may be
potentially useful(?) for inspection/archive/debugging/reuse later, so
keeping them completely hidden may not be desired.

These files are deleted upon successful completion of the module using the
iostream lib, nothing left to inspect. They are probably only useful for
the iostream lib developers for debugging. Same for the segment lib,
intermediate files are probably only useful for developers working on the
segment lib. I don't see a reason why files created by the iostream
library should be treated differently to files created by the segment
library, or vice versa, when it comes to their storage location.

This discussion is now moving away from r.terraflow towards where to store
large temporary files, maybe this ticket can be closed and the discussion
be continued somewhere else if needed?

Markus M

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/520#comment:20&gt;
GRASS GIS <http://grass.osgeo.org>

22

#520: r.terraflow/iostream does not respect STREAM_DIR option
---------------------+------------------------------------------------------
  Reporter: mmetz | Owner: grass-dev@lists.osgeo.org
      Type: defect | Status: new
  Priority: major | Milestone: 6.4.0
Component: Raster | Version: svn-develbranch6
Resolution: | Keywords: r.terraflow, tmp
  Platform: All | Cpu: All
---------------------+------------------------------------------------------
Comment (by neteler):

Replying to [comment:20 mmetz]:
> Replying to [comment:19 hamish]:
> > Replying to [comment:18 mmetz]:
> > ISTR that the Terraflow team (Andrew? Laura?) explained why they used
that a long time ago (I now forget exact reason),

Most likely because r.terraflow wasn't originally written for GRASS/was
written also as stand-alone version.

> Picking a directory whose contents are deleted automatically if I
remember right, which would be the case for $MAPSET/.tmp/$MACHINE.

Perfect. I assume that 99% of the users will never inspect any tmp file of
r.terraflow. So that is a perfect default. If a specialist wants to
inspect, then define a different directory +in that case*.

> > Note that these temp files may be >2gig
>
> I get regularly >20GB, having to work on somewhat larger datasets.
Thinking about Doug's datasets, it would go way above 100GB.

Yes, and I always forget to look in /var/tmp/ for undeleted rubbish (but
check instead my HOME for delete candidates). Having auto-deletion of tmp
stuff left by interrupted/crashed r.terraflow runs when leaving GRASS
would be a major advantage for most users.

> > (use the memory= option to lower those needs), and they may be
potentially useful(?) for inspection/archive/debugging/reuse later, so
keeping them completely hidden may not be desired.

As said, who wants to analyse these files is also able to change a default
setting.

> These files are deleted upon successful completion of the module using
the iostream lib, nothing left to inspect. They are probably only useful
for the iostream lib developers for debugging. Same for the segment lib,
intermediate files are probably only useful for developers working on the
segment lib. I don't see a reason why files created by the iostream
library should be treated differently to files created by the segment
library, or vice versa, when it comes to their storage location.

+1

> This discussion is now moving away from r.terraflow towards where to
store large temporary files, maybe this ticket can be closed and the
discussion be continued somewhere else if needed?

No, still /var/tmp/ is the default also in GRASS 7. This should be
changed.

MarkusN

> Markus M

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/520#comment:21&gt;
GRASS GIS <http://grass.osgeo.org>

23

These files are deleted upon successful completion of the module using

the iostream lib, nothing left to inspect. They are probably only useful
for the iostream lib developers for debugging. Same for the segment lib,
intermediate files are probably only useful for developers working on the
segment lib. I don't see a reason why files created by the iostream
library should be treated differently to files created by the segment
library, or vice versa, when it comes to their storage location.

+1

This discussion is now moving away from r.terraflow towards where to

store large temporary files, maybe this ticket can be closed and the
discussion be continued somewhere else if needed?

No, still /var/tmp/ is the default also in GRASS 7. This should be
changed.

yes, please, this needs to be changed, it has been a long standing issue

Helena

MarkusN

Markus M

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/520#comment:21&gt;
GRASS GIS <http://grass.osgeo.org>
_______________________________________________
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

24

#520: r.terraflow/iostream does not respect STREAM_DIR option
------------------------------+---------------------------------------------
Reporter: mmetz | Owner: grass-dev@…
     Type: defect | Status: new
Priority: major | Milestone: 6.4.3
Component: Raster | Version: svn-releasebranch64
Keywords: r.terraflow, tmp | Platform: All
      Cpu: All |
------------------------------+---------------------------------------------
Changes (by neteler):

  * version: svn-develbranch6 => svn-releasebranch64
  * milestone: 6.4.0 => 6.4.3

Comment:

We still have /var/tmp/ as default answer instead of
$MAPSET/.tmp/$MACHINE.
This should be updated (some g.parser magic will be needed).

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/520#comment:22&gt;
GRASS GIS <http://grass.osgeo.org>

25

#520: r.terraflow/iostream does not respect STREAM_DIR option
---------------------+------------------------------------------------------
  Reporter: mmetz | Owner: grass-dev@…
      Type: defect | Status: closed
  Priority: major | Milestone: 6.4.3
Component: Raster | Version: svn-releasebranch64
Resolution: fixed | Keywords: r.terraflow, tmp
  Platform: All | Cpu: All
---------------------+------------------------------------------------------
Changes (by mmetz):

  * status: new => closed
  * resolution: => fixed

Comment:

Replying to [comment:22 neteler]:
> We still have /var/tmp/ as default answer instead of
$MAPSET/.tmp/$MACHINE.
> This should be updated (some g.parser magic will be needed).

Fixed in r52749/50.

Markus M

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/520#comment:23&gt;
GRASS GIS <http://grass.osgeo.org>