#3309: Avoid errors in parallel creation of tempfiles
--------------------------+---------------------------------
Reporter: sbl | Owner: grass-dev@…
Type: enhancement | Status: new
Priority: normal | Milestone: 7.2.1
Component: LibGIS | Version: svn-releasebranch72
Keywords: G_tempfile() | CPU: Unspecified
Platform: Unspecified |
--------------------------+---------------------------------
When tempfiles are created in parallel, conflicts and thus fatal errors
can arise when two processes try to create the .tmp/HOST dir at the same
time: https://lists.osgeo.org/pipermail/grass-dev/2017-March/084471.html
The conflict seems to arise only from the directory.
So, maybe it could help if the directory is created at the start of a
GRASS session and removed only when the GRASS session is closed (exit)?
#3309: Avoid errors in parallel creation of tempfiles
--------------------------+---------------------------------
Reporter: sbl | Owner: grass-dev@…
Type: enhancement | Status: new
Priority: normal | Milestone: 7.2.1
Component: LibGIS | Version: svn-releasebranch72
Resolution: | Keywords: G_tempfile()
CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------
Comment (by mlennert):
Replying to [ticket:3309 sbl]:
> When tempfiles are created in parallel, conflicts and thus fatal errors
can arise when two processes try to create the .tmp/HOST dir at the same
time:
> https://lists.osgeo.org/pipermail/grass-dev/2017-March/084471.html
>
> The conflict seems to arise only from the directory.
>
> So, maybe it could help if the directory is created at the start of a
GRASS session and removed only when the GRASS session is closed (exit)?
Now that you mention it, I can confirm that I've also been confronted to
such errors when running commands in parallel that create temp files. So
+1 to needing a solution.
I don't know though if the best is to create this directory once and for
all at the startup, or whether the tempfile routines in lib/gis/tempfile.c
could be modified to avoid such race conditions.
#3309: Avoid errors in parallel creation of tempfiles
--------------------------+---------------------------------
Reporter: sbl | Owner: grass-dev@…
Type: enhancement | Status: new
Priority: normal | Milestone: 7.2.1
Component: LibGIS | Version: svn-releasebranch72
Resolution: | Keywords: G_tempfile()
CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------
Comment (by mmetz):
Replying to [comment:1 mlennert]:
> Replying to [ticket:3309 sbl]:
> > When tempfiles are created in parallel, conflicts and thus fatal
errors can arise when two processes try to create the .tmp/HOST dir at the
same time:
> > https://lists.osgeo.org/pipermail/grass-dev/2017-March/084471.html
> >
> > The conflict seems to arise only from the directory.
> >
> > So, maybe it could help if the directory is created at the start of a
GRASS session and removed only when the GRASS session is closed (exit)?
>
> Now that you mention it, I can confirm that I've also been confronted to
such errors when running commands in parallel that create temp files. So
+1 to needing a solution.
Strange. When starting GRASS in text or GUI mode, the .tmp/HOST folder is
automatically created, i.e. the error
{{{
ERROR: Unable to make mapset element .tmp/HOST
(/grassdata/ETRS_33N/timseries/.tmp): File exists
}}}
should not happen, at least not in trunk and relb72.
>
> I don't know though if the best is to create this directory once and for
all at the startup, or whether the tempfile routines in lib/gis/tempfile.c
could be modified to avoid such race conditions.
You could use
{{{
rm -f "`g.tempfile pid=$$`"
}}}
before running xargs, just to make sure that .tmp/HOST has been created.
#3309: Avoid errors in parallel creation of tempfiles
--------------------------+---------------------------------
Reporter: sbl | Owner: grass-dev@…
Type: enhancement | Status: new
Priority: normal | Milestone: 7.2.1
Component: LibGIS | Version: svn-releasebranch72
Resolution: | Keywords: G_tempfile()
CPU: Unspecified | Platform: Unspecified
--------------------------+---------------------------------
Comment (by sbl):
For the record:
Running
{{{
rm -f "`g.tempfile pid=$$`"
}}}
in advance works around the issue.
Using GRASS 7.2.1svn (r70188) in BATCH mode I do get the error message
above. But if I run g.tempfile and make sure that the tmp directory is
created before I run r.in.gdal through xargs, everything works just fine!
Replying to [comment:9 neteler]:
> Does this issue persist in GRASS GIS 7.6.svn?
Please, if you can, test trunk (7.7) too - a lot of changes to temporary
directory handling (#613, #1286, #3635, ...). And if possible, please,
test also `GRASS_BATCH_JOB` versus `--exec`.