So I think a cron job, possibly running under Martin L's user went awry
on projects and filled 30% of the disk in hours, filling the disk
completely.
Can someone more familiar with what these jobs are supposed to do,
please kill the job, cleanup the excess files and figure out how to
prevent future incidents.
FYI, while we may have finished replacing disks and batteries it seems
that too frequent building of docs is still taking a toll on disk i/o.
We should discuss this further in the 2014 Infrastructure proposal.
So I think a cron job, possibly running under Martin L's user went awry
on projects and filled 30% of the disk in hours, filling the disk
completely.
Can someone more familiar with what these jobs are supposed to do,
please kill the job, cleanup the excess files and figure out how to
prevent future incidents.
It was a doc build cron job for grass6 addon modules which got stuck in a recursive loop re-adding the text of the page to a temp file over and over until the disk filled. The job is now killed, the massive .tmp.html file deleted, and apache restarted.
Hopefully the cause of the problem can be figured out before it happens again.
Right now the cron job is still enabled, if we (i.e. grass-dev) don't figure out the cause in <24 hrs I'd suggest to disable the cron job while we figure out a solution and an alternate place to run lower priority jobs, and the web server's cache can remain in memory longer to lessen the disk i/o issues.
FYI, while we may have finished replacing disks and batteries it seems
that too frequent building of docs is still taking a toll on disk i/o.
We should discuss this further in the 2014 Infrastructure proposal.
I look forward to a dedicated build server, and knowing the load it will have to cope with, I wish it luck.
On Fri, Apr 25, 2014 at 5:38 AM, Alex Mandel <tech_dev@wildintellect.com> wrote:
So I think a cron job, possibly running under Martin L's user went awry
on projects and filled 30% of the disk in hours, filling the disk
completely.
I am checking but don't see anything: 56G available.
Where did you see that?
...
FYI, while we may have finished replacing disks and batteries it seems
that too frequent building of docs is still taking a toll on disk i/o.
We should discuss this further in the 2014 Infrastructure proposal.
The performance is still a fraction as it used to be. Even in this
moment when the machine only servers Web pages and no cronjob running
at all... Something is still odd/different to what it was some few
weeks before.
