Hi,
it seems that sometimes problems occur with the cleaning of
temporal files. Normally, when a GRASS session is closed,
the tool
$GISBASE/etc/clean_temp
is called. However, people who work in a single project
may not close the GRASS session for weeks. In that
case tons of temporal data in
$LOCATION/$MAPSET/.tmp/$HOSTNAME/
are stored, consuming eventually lot of space (I just discovered
temp files totally consuming 1GB in our network).
Does anyone have a suggestion if we can install something
like a cronjob or whatever, which deletes temp files of
a certain age?
Markus
Markus Neteler wrote:
it seems that sometimes problems occur with the cleaning of
temporal files. Normally, when a GRASS session is closed,
the tool
$GISBASE/etc/clean_temp
is called. However, people who work in a single project
may not close the GRASS session for weeks. In that
case tons of temporal data in
$LOCATION/$MAPSET/.tmp/$HOSTNAME/
are stored, consuming eventually lot of space (I just discovered
temp files totally consuming 1GB in our network).
Does anyone have a suggestion if we can install something
like a cronjob or whatever, which deletes temp files of
a certain age?
Are you referring to having GRASS do this by default (as opposed to
something specific to your site)? If so, then it is complicated by the
fact that "crontab" commands aren't entirely portable (there's at
least POSIX syntax, SVR3 syntax and the old Vixie-cron syntax).
Also, you may need to create separate crontab entries for each user.
If you run a "temp cleaner" as root, you need to allow for the
possibility of malicious users tricking it into deleting arbitrary
files through a combination of symbolic links and race conditions.
--
Glynn Clements <glynn.clements@virgin.net>
Markus Neteler wrote:
Does anyone have a suggestion if we can install something
like a cronjob or whatever, which deletes temp files of
a certain age?
One other point related to temp files; the command which d.text
registers with D_add_to_list refers to a temporary file which contains
the text:
sprintf(max_buff, "%s < %s", G_recreate_command(), wind_file_name);
D_add_to_list(max_buff);
The only alternative would be to change the interface to d.text so
that the text was either given as a command-line argument, or in
stored in a named file which is given as a command-line argument.
This is a symptom of a more general problem with concept behind
D_add_to_list(), namely that a command which runs initially may fail
or behave differently when run at a later time.
--
Glynn Clements <glynn.clements@virgin.net>
On Wed, Apr 17, 2002 at 12:35:50PM +0100, Glynn Clements wrote:
Markus Neteler wrote:
> Does anyone have a suggestion if we can install something
> like a cronjob or whatever, which deletes temp files of
> a certain age?
What about a command to explicitly call cleanup.
Then users could issue this manually.
They might even set up a cron job themselfs.
On Wed, Apr 17, 2002 at 12:28:32PM +0100, Glynn Clements wrote:
Markus Neteler wrote:
> it seems that sometimes problems occur with the cleaning of
> temporal files. Normally, when a GRASS session is closed,
> the tool
> $GISBASE/etc/clean_temp
>
> is called. However, people who work in a single project
> may not close the GRASS session for weeks. In that
> case tons of temporal data in
> $LOCATION/$MAPSET/.tmp/$HOSTNAME/
> are stored, consuming eventually lot of space (I just discovered
> temp files totally consuming 1GB in our network).
>
> Does anyone have a suggestion if we can install something
> like a cronjob or whatever, which deletes temp files of
> a certain age?
[thanks for the d.text pointer]
Are you referring to having GRASS do this by default (as opposed to
something specific to your site)? If so, then it is complicated by the
fact that "crontab" commands aren't entirely portable (there's at
least POSIX syntax, SVR3 syntax and the old Vixie-cron syntax).
Also, you may need to create separate crontab entries for each user.
If you run a "temp cleaner" as root, you need to allow for the
possibility of malicious users tricking it into deleting arbitrary
files through a combination of symbolic links and race conditions.
I would prefer to have the current GRASS session taking care of
the cleaning. That keeps maintenance at a minimum (we have often
students here for short time, also people tend to login on
different machines and access the centralized location etc.).
Maybe the GRASS-daemon Roger is developing can watch the session and
delete things if unnecessary. Or some 'at' job started after opening
a new GRASS session?
Markus
On Wed, Apr 17, 2002 at 01:44:26PM +0200, Bernhard Reiter wrote:
On Wed, Apr 17, 2002 at 12:35:50PM +0100, Glynn Clements wrote:
> Markus Neteler wrote:
>
> > Does anyone have a suggestion if we can install something
> > like a cronjob or whatever, which deletes temp files of
> > a certain age?
What about a command to explicitly call cleanup.
Then users could issue this manually.
They might even set up a cron job themselfs.
We are talking about users 
They won't do it, maybe only the first day, then they forget about
that.
So it must be automated.
Markus
On Wed, Apr 17, 2002 at 01:52:11PM +0200, Markus Neteler wrote:
On Wed, Apr 17, 2002 at 01:44:26PM +0200, Bernhard Reiter wrote:
> On Wed, Apr 17, 2002 at 12:35:50PM +0100, Glynn Clements wrote:
> > Markus Neteler wrote:
> >
> > > Does anyone have a suggestion if we can install something
> > > like a cronjob or whatever, which deletes temp files of
> > > a certain age?
>
> What about a command to explicitly call cleanup.
> Then users could issue this manually.
>
> They might even set up a cron job themselfs.
We are talking about users
They won't do it, maybe only the first day, then they forget about
that.
So it must be automated.
They'll get a warning about their quota then.
A system admin also could do it.
Another idea would be to add a hook for a check when
other temp files are created.
On Wed, Apr 17, 2002 at 01:47:00PM +0200, Markus Neteler wrote:
On Wed, Apr 17, 2002 at 12:28:32PM +0100, Glynn Clements wrote:
Maybe the GRASS-daemon Roger is developing can watch the session and
delete things if unnecessary. Or some 'at' job started after opening
a new GRASS session?
The daemon certainly is nothing for 5.0. and probably not even for 5.2.
Bernhard
On Wed, Apr 17, 2002 at 01:56:11PM +0200, Bernhard Reiter wrote:
On Wed, Apr 17, 2002 at 01:52:11PM +0200, Markus Neteler wrote:
> On Wed, Apr 17, 2002 at 01:44:26PM +0200, Bernhard Reiter wrote:
> > On Wed, Apr 17, 2002 at 12:35:50PM +0100, Glynn Clements wrote:
> > > Markus Neteler wrote:
> > >
> > > > Does anyone have a suggestion if we can install something
> > > > like a cronjob or whatever, which deletes temp files of
> > > > a certain age?
> >
> > What about a command to explicitly call cleanup.
> > Then users could issue this manually.
> >
> > They might even set up a cron job themselfs.
>
> We are talking about users
> They won't do it, maybe only the first day, then they forget about
> that.
>
> So it must be automated.
They'll get a warning about their quota then.
We do not use quota here for various reasons. Even then, most users
would not understand the reason (as the .tmp/ directory is rather hidden).
A system admin also could do it.
If he/she finds free time
The problem is that I regularly remind people to remove left-over temp files
from GRASS. This looks like a conceptional weakness in 'clean_temp'.
When working in remote sensing, you can quickly flood your machine without
realizing why (especially non-experienced users). That's why I suggest an
improvement here which is GRASS driven.
Another idea would be to add a hook for a check when
other temp files are created.
Markus
Markus Neteler wrote:
Maybe the GRASS-daemon Roger is developing can watch the session and
delete things if unnecessary. Or some 'at' job started after opening
a new GRASS session?
Well, "at" only runs the command once, but you could have it run a
script which runs "at" to re-install itself.
--
Glynn Clements <glynn.clements@virgin.net>
On Wednesday 17 April 2002 06:12, Glynn Clements wrote:
Markus Neteler wrote:
> Maybe the GRASS-daemon Roger is developing can watch the session and
> delete things if unnecessary. Or some 'at' job started after opening
> a new GRASS session?
Well, "at" only runs the command once, but you could have it run a
script which runs "at" to re-install itself.
The daemon could be made to clean up temporary files. Currently the daemon
works only in response to user-generated events so it would be something of
an expansion of the concept, but not a big one.
Another option would be to modify a library routine (G_tempfile perhaps) so
that it checks for and cleans unused temp files whenever it is invoked. That
would not keep .tmp completely clean but it would keep the temp files from
building up.
A note about the daemon. It it working very well. I installed it over the
most recent tarball I downloaded. The updates to make it work were required
by changes in GRASS since pre2, not by problems in the daemon code itself. I
still need to track down the cause of an odd error generated by clean_temp on
exit. I had a problem with location formats that I also need to track down,
but that may have been caused by earlier troubles. The last step will be to
figure out how and where to install it in the GRASS hierarchy.
Roger Miller
Markus Neteler wrote:
Does anyone have a suggestion if we can install something
like a cronjob or whatever, which deletes temp files of
a certain age?
That can be done, but is only a workaround : maybe the real
question is "why does Grass spread temp files all around ?"
Any command that terminates should erase the various temp files
created.
Even when things are going bad, there are many ways to prevent
the file to be left (the most used is to delete the file just
after its creation : it will only disapears once closed, and
that is automaticaly done by the system when the command
terminates).
If only temp files from scripts (history, display lists, ...)
remain, they would not be too big and the clean_temp function
is enough...
--
Michel WURTZ - DIG - Maison de la télédétection
500, rue J.F. Breton
34093 MONTPELLIER Cedex 5
On Thu, Apr 18, 2002 at 02:50:11PM +0000, Michel Wurtz wrote:
Markus Neteler wrote:
>
> Does anyone have a suggestion if we can install something
> like a cronjob or whatever, which deletes temp files of
> a certain age?
That can be done, but is only a workaround : maybe the real
question is "why does Grass spread temp files all around ?"
Any command that terminates should erase the various temp files
created.
A good point! It seems that we have one or more leackages here.
Question is if we could add something to the temp file creation
routine G_tempfile() or so to find out the generating function
from the temp file. Maybe
8812.i.rectify
or so? Then I could watch the temp directory a while and
easily identify the candidates.
Even when things are going bad, there are many ways to prevent
the file to be left (the most used is to delete the file just
after its creation : it will only disapears once closed, and
that is automaticaly done by the system when the command
terminates).
If only temp files from scripts (history, display lists, ...)
remain, they would not be too big and the clean_temp function
is enough...
The problem is that clean_temp is only called when leaving GRASS.
If you stay in a session for one week or month (yes, that happens),
you just fill the temp space.
Markus