[GRASS-dev] [GRASS GIS] #2185: Painfully Slow 'v.in.ogr' Vector Import

GDAL · February 1, 2014, 8:39pm

#2185: Painfully Slow 'v.in.ogr' Vector Import
--------------------------------------+-------------------------------------
Reporter: justinzane | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Keywords: import, OGR, performance | Platform: Linux
Cpu: Unspecified |
--------------------------------------+-------------------------------------
I'm trying to import shapefiles (in this case, OpenStreetMap data) into
GRASS (grass70-svn) using `v.in.ogr`; and it is '''painfully, brutally
slow'''. By slow, I mean between 13 and 28 ''primitives per second''.

At first I thought this was because I was using the default sqlite
database on an NFS mount. But, trying sqlite on a local disk and trying
MySQL to separate server both failed to improve performance. I'm getting a
consistent read/write of ~125Kbps/~550Kbps no matter which shapefile or
which database I use.

I cannot see any bottlenecks anywhere, as the two `python /path/to/grass-
gui` processes are averaging 10% of one core combined, the network
utilization is below 10% consistently, and the DB server is almost
quiescent. To give an example of how painful this is:

     v.in.ogr dsn=/home/justin/downloads/osm_CA/places.shp
     WARNING: All available OGR layers will be imported into vector map
<places>
     Check if OGR layer <places> contains polygons...
     Importing 7020 features (OGR layer <places>)...
     -----------------------------------------------------
     Building topology for vector map <places@OSM>...
     Registering primitives...
     7020 primitives registered
     7020 vertices registered
     Building areas...
     0 areas built
     0 isles built
     Attaching islands...
     Attaching centroids...
     Number of nodes: 0
     Number of primitives: 7020
     Number of points: 7020
     Number of lines: 0
     Number of boundaries: 0
     Number of centroids: 0
     Number of areas: 0
     Number of isles: 0
     (Fri Jan 31 19:01:08 2014) Command finished (4 min 38 sec)

Note that this may belong under Database.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2185>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 2, 2014, 7:12am

#2185: Painfully Slow 'v.in.ogr' Vector Import
--------------------------------------+-------------------------------------
Reporter: justinzane | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Keywords: import, OGR, performance | Platform: Linux
Cpu: Unspecified |
--------------------------------------+-------------------------------------
Changes (by justinzane):

* cc: justinzane (added)

Comment:

Running under `perf` several times, and using `-Og -g` builds of grass7
and gdal, it seems that somewhere between 30-40% of the execution time is
happening in Python or the shell. That seems odd, but I'm ignorant...

Additionally, it seems that grass is sending a query for **every** write
to the DB, **synchronously**, instead of either doing the DB IO in another
thread. That seems a poor use of multicored systems.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2185#comment:1>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 3, 2014, 9:20am

#2185: Painfully Slow 'v.in.ogr' Vector Import
--------------------------------------+-------------------------------------
Reporter: justinzane | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Keywords: import, OGR, performance | Platform: Linux
Cpu: Unspecified |
--------------------------------------+-------------------------------------

Comment(by mmetz):

Replying to [ticket:2185 justinzane]:
> I'm trying to import shapefiles (in this case, OpenStreetMap data) into
GRASS (grass70-svn) using `v.in.ogr`; and it is '''painfully, brutally
slow'''. By slow, I mean between 13 and 28 ''primitives per second''.

>
> I cannot see any bottlenecks anywhere, as the two `python /path/to
/grass-gui` processes are averaging 10% of one core combined, the network
utilization is below 10% consistently, and the DB server is almost
quiescent. To give an example of how painful this is:
>
{{{
      v.in.ogr dsn=/home/justin/downloads/osm_CA/places.shp
      WARNING: All available OGR layers will be imported into vector map
<places>
      Check if OGR layer <places> contains polygons...
      Importing 7020 features (OGR layer <places>)...
      -----------------------------------------------------
      Building topology for vector map <places@OSM>...
      Registering primitives...
      7020 primitives registered
      7020 vertices registered
      Building areas...
      0 areas built
      0 isles built
      Attaching islands...
      Attaching centroids...
      Number of nodes: 0
      Number of primitives: 7020
      Number of points: 7020
      Number of lines: 0
      Number of boundaries: 0
      Number of centroids: 0
      Number of areas: 0
      Number of isles: 0
      (Fri Jan 31 19:01:08 2014) Command finished (4 min 38 sec)
}}}
>
On a standard laptop with default settings, the import of points is about
140 x faster. 7000 points from a shapefile should be imported in less than
2 seconds.

You would need to monitor the `v.in.ogr` process and the corresponding db
driver process (`sqlite` for the default sqlite database in GRASS 7)
instead of any Python processes and see if `v.in.ogr` uses close to 100%
CPU. If not, `sqlite` should be very busy, but that should not happen
because all changes to the database are committed at once after importing
the primitives.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2185#comment:2>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 4, 2014, 12:25am

#2185: Painfully Slow 'v.in.ogr' Vector Import
--------------------------------------+-------------------------------------
Reporter: justinzane | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Keywords: import, OGR, performance | Platform: Linux
Cpu: x86-64 |
--------------------------------------+-------------------------------------
Changes (by justinzane):

* cpu: Unspecified => x86-64

Comment:

Thanks for the response. I cannot post the dot graphs because of size
restrictions, and resizing them makes them near unreadable.

I did another little experiement, and, it seems that the problem is
**not** with the database driver at all. WRT MySQL and SQLite, it is the
location of the 'grassdata' folder that is at issue. I had kept
'grassdata' on an NFS mount. Moving it to a local (btrfs) spinning disk
meant that the import was **many** times faster (although moron forgot to
add 'time' before the command ).

I'm rebuilding an unstripped glibc and other dependencies to see exactly
which calls may be problematic.

As far as monitoring the mysql and v.in.ogr processes, do you have any
recommended way besides running grass like:

{{{
perf record -e cycles:u -o grass.prof -g -- grass64 -text
/home/justin/maps/grassdata/osm_data/OSM;
####
# In grass shell [zsh]:
# v.in.ogr dsn=./path/to/shape.shp output=shape
# exit
####
perf script -i grass.prof | gprof2dot -f perf | dot -Tpng -o output.png;
gwenview output.png
}}}

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2185#comment:3>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 4, 2014, 1:19am

#2185: Painfully Slow 'v.in.ogr' Vector Import
--------------------------------------+-------------------------------------
Reporter: justinzane | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Keywords: import, OGR, performance | Platform: Linux
Cpu: x86-64 |
--------------------------------------+-------------------------------------

Comment(by wenzeslaus):

Replying to [comment:3 justinzane]:
> Thanks for the response. I cannot post the dot graphs because of size
restrictions, and resizing them makes them near unreadable.
>
Curiosity: Can you post the `.dot` file? Is it small enough, e.g. 7ziped
or after using `gprof2dot`'s `--node-thres` or `--edge-thres`?

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2185#comment:4>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 4, 2014, 1:24am

#2185: Painfully Slow 'v.in.ogr' Vector Import
--------------------------------------+-------------------------------------
Reporter: justinzane | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Keywords: import, OGR, performance | Platform: Linux
Cpu: x86-64 |
--------------------------------------+-------------------------------------

Comment(by justinzane):

I'll work on it. I'm rebuilding a few things now [and getting wedding pics
ready to send to my niece, so dcraw is hogging the cpu and net] so as soon
as I can do a couple of comparisons for you I will. I'll post them to my
webserver if they are too large, and just add the links.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2185#comment:5>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 4, 2014, 7:28pm

#2185: Painfully Slow 'v.in.ogr' Vector Import
--------------------------------------+-------------------------------------
Reporter: justinzane | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Keywords: import, OGR, performance | Platform: Linux
Cpu: x86-64 |
--------------------------------------+-------------------------------------

Comment(by justinzane):

Please see [http://www.justinzane.com/grass-perf.html\] for some more
detailed data. I'm greatly interested in guidance on narrowing down the
bottleneck with "grassdata" on a NFS mount, but I may need some assistance
in the process. If anyone is on IRC, I'd be more than happy to work there.
Thanks.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2185#comment:6>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 4, 2014, 9:30pm

#2185: Painfully Slow 'v.in.ogr' Vector Import
--------------------------------------+-------------------------------------
Reporter: justinzane | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Keywords: import, OGR, performance | Platform: Linux
Cpu: x86-64 |
--------------------------------------+-------------------------------------

Comment(by mmetz):

Replying to [comment:6 justinzane]:
> I'm greatly interested in guidance on narrowing down the bottleneck with
"grassdata" on a NFS mount, but I may need some assistance in the process.

I don't know much about perf, but I do have some experience with accessing
a GRASS database over NFS (multiple clients accessing the same GRASS
database over NFS at the same time). The problem seems to be not the GRASS
+ NFS combination, but the connection to the NFS server (should be at
least a Gigabit connection and static IP addresses for the server and all
its clients) and other traffic simultaneously going to the NFS server from
other clients.

You could
  * check the IO traffic on the NFS server without using GRASS
  * copy some stuff, e.g. an unpacked Linux kernel, from the local disk
over to the NFS mount and check the performance
  * investigate the actually used NFS mount options in /proc/mounts
  * try to tune NFS mount options, particularly rsize,wsize, bigger seems
to be better, more in `man nfs`

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2185#comment:7>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 4, 2014, 10:11pm

#2185: Painfully Slow 'v.in.ogr' Vector Import
--------------------------------------+-------------------------------------
Reporter: justinzane | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Keywords: import, OGR, performance | Platform: Linux
Cpu: x86-64 |
--------------------------------------+-------------------------------------

Comment(by justinzane):

[quote]check the IO traffic on the NFS server without using GRASS[/quote]
Did, using iotop while running v.in.ogr -- No noticeable IO besides my
GRASS import, which is expected since it is a personal server.

[quote]copy some stuff, e.g. an unpacked Linux kernel, from the local disk
over to the NFS mount and check the performance[/quote]
Works swimmingly for large files. I updated the post at
[www.justinzane.com/grass-perf.html] to show a simplistic test. Basically,
though my Dell laptop has a castrated NIC -- the 4 gigabit pins are not
connected! -- NFS works at almost line speed. Throughput is not the
problem, rather it seems to be excessive opens/syncs/similar that is the
issue.

Using `nfsstat -Z10`, I get about 27 writes per second. Which seems to
equate to one write per primitive! I'm a GIS novice, but it seems that
writing for every primitive is not efficient from a programming
perspective. Am I missing something?

[quote]investigate the actually used NFS mount options in
/proc/mounts[/quote]
{{{nfs4
rw,relatime,vers=4.0,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.182,local_lock=none,addr=192.168.0.10
0 0}}}

[quote]try to tune NFS mount options, particularly rsize,wsize, bigger
seems to be better, more in man nfs [/quote]
With writes of a few bytes to a few kB, this does not matter.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2185#comment:8>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 5, 2014, 11:35am

#2185: Painfully Slow 'v.in.ogr' Vector Import
--------------------------------------+-------------------------------------
Reporter: justinzane | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Keywords: import, OGR, performance | Platform: Linux
Cpu: x86-64 |
--------------------------------------+-------------------------------------

Comment(by mmetz):

I did some tests with and without NFS, importing a shapefile with 50000
points and 19 attribute fields:
  * shapefile on local disk, grassdata on local disk: 12000 points / sec
  * shapefile on local disk, grassdata on NFS: 2200 points / sec
  * shapefile on NFS, grassdata on NFS: 2100 points / sec
  * shapefile on NFS, grassdata on local disk: 11000 points / sec

Here, writing to grassdata on NFS is also much slower, but always much
faster than about 20 points / sec. The NFS server here is not at all a
high-performance server, instead designed for power-saving.

Your minifstest gives here

{{{
NumChars: Local (s) NFS (s) NFS/Local
00000002: 000.0100, 000.0200, 002.0000
00000004: 000.0100, 000.0200, 002.0000
00000008: 000.0200, 000.0300, 001.5000
00000016: 000.0200, 000.0400, 002.0000
00000032: 000.0400, 000.0400, 001.0000
00000128: 000.1300, 000.1600, 001.2308
00000512: 000.5200, 000.5300, 001.0192
}}}

The penalty for doing lots of tiny open-write-close cycles on NFS is here
maximum 2, not 7.

IMHO, something is strange with your NFS server, but I have no idea what.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2185#comment:9>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 5, 2014, 11:03pm

#2185: Painfully Slow 'v.in.ogr' Vector Import
--------------------------------------+-------------------------------------
Reporter: justinzane | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Keywords: import, OGR, performance | Platform: Linux
Cpu: x86-64 |
--------------------------------------+-------------------------------------

Comment(by justinzane):

I noticed that. There are a bunch of nfs/nfs4 perf events; **but**, they
require root privileges when running perf. I'm trying to get advice
(elsewhere) on how to properly use them without running GRASS as root. As
soon as I am able to examine the NFS behaviour, I'll post the results. I'm
also pulling the code so I can speak with less ignorance.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2185#comment:10>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 6, 2014, 4:07am

#2185: Painfully Slow 'v.in.ogr' Vector Import
------------------------------------------------+---------------------------
Reporter: justinzane | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Keywords: import, OGR, performance, v.in.ogr | Platform: Linux
Cpu: x86-64 |
------------------------------------------------+---------------------------
Changes (by hamish):

* keywords: import, OGR, performance => import, OGR, performance,
v.in.ogr

Comment:

@justinzane: if you get a few moments could you add a quick example on our
wiki of how end users can run `pref`? tx.
http://grasswiki.osgeo.org/wiki/Bugs#Using_a_profiling_tool

@mmetz: some sample OSM data to test is available here:
http://download.osgeo.org/livedvd/data/osm/

I'm still not understanding if this trouble is CPU bound, local IO bound,
or network bound. Some combo of top, iotop, iftop, and gkrellm would
hopefully shed some light on that without having to push everything
through a profiling tool. What does 'top -i' show about the v.in.ogr and
sqlite processes and kernel %use state?
is it dead locked? spending all its time in the kernel? libogr..?

regards,
Hamish

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2185#comment:11>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 6, 2014, 4:18am

#2185: Painfully Slow 'v.in.ogr' Vector Import
------------------------------------------------+---------------------------
Reporter: justinzane | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Keywords: import, OGR, performance, v.in.ogr | Platform: Linux
Cpu: x86-64 |
------------------------------------------------+---------------------------

Comment(by hamish):

btw, you might also try osm2pgsql to load the OSM data into a PostGIS
Postgres DB, then live-connect to the PostGIS db with grass 7 with
v.external. for example:

https://trac.osgeo.org/osgeo/browser/livedvd/gisvm/trunk/bin/load_postgis.sh#L85

otherwise map coordinates are stored in $MAPSET/vector in grass's vector
format, and only feature attributes are stored in the DB. Map features are
connected to attributes via category number.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2185#comment:12>
GRASS GIS <http://grass.osgeo.org>

GDAL · January 29, 2015, 1:26pm

#2185: Painfully Slow 'v.in.ogr' Vector Import
------------------------------------------------+---------------------------
Reporter: justinzane | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Keywords: import, OGR, performance, v.in.ogr | Platform: Linux
Cpu: x86-64 |
------------------------------------------------+---------------------------

Comment(by imincik1):

I am experiencing this issue too. It seems to be a problem with all
modules which are writing simple vector geometries in loop. I am going to
investigate some sqlite tweeks [1].

1 - http://www.sqlite.org/faq.html#q19

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:13>
GRASS GIS <http://grass.osgeo.org>

GDAL · January 29, 2015, 11:34pm

#2185: Painfully Slow 'v.in.ogr' Vector Import
------------------------------------------------+---------------------------
Reporter: justinzane | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Keywords: import, OGR, performance, v.in.ogr | Platform: Linux
Cpu: x86-64 |
------------------------------------------------+---------------------------

Comment(by imincik1):

In my situation, I have whole home directory including GRASS data dir on
NFS 4 mount.

1. After short investigation, it seems, that setting PRAGMA
synchronous=OFF on SQLite db has no effect.

2. What was a great change was moving '.tmp' directory from my mapset
outside of NFS (by using symlink). v.in.ogr was 8 times faster.
So my idea is, is it possible to have some environment variable, which
could define global alternative directory for temporary files ? This could
improve situation for all modules which are using temporary files (for
example v.random is not using).

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:14>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 1, 2015, 5:34pm

#2185: Painfully Slow 'v.in.ogr' Vector Import
------------------------------------------------+---------------------------
Reporter: justinzane | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Keywords: import, OGR, performance, v.in.ogr | Platform: Linux
Cpu: x86-64 |
------------------------------------------------+---------------------------

Comment(by neteler):

Replying to [comment:14 imincik1]:
...
> 2. What was a great change was moving '.tmp' directory from my mapset
outside of NFS (by using symlink). v.in.ogr was 8 times faster.
> So my idea is, is it possible to have some environment variable, which
could define global alternative directory for temporary files ? This could
improve situation for all modules which are using temporary files (for
example v.random is not using).

Can you please try this variable?
http://grass.osgeo.org/grass70/manuals/variables.html

--> "TMPDIR, TEMP, TMP" (name depends on the operating system)

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:15>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 1, 2015, 8:49pm

#2185: Painfully Slow 'v.in.ogr' Vector Import
------------------------------------------------+---------------------------
Reporter: justinzane | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Keywords: import, OGR, performance, v.in.ogr | Platform: Linux
Cpu: x86-64 |
------------------------------------------------+---------------------------

Comment(by imincik1):

Replying to [comment:15 neteler]:
> Replying to [comment:14 imincik1]:
> ...
> > 2. What was a great change was moving '.tmp' directory from my mapset
outside of NFS (by using symlink). v.in.ogr was 8 times faster.
> > So my idea is, is it possible to have some environment variable, which
could define global alternative directory for temporary files ? This could
improve situation for all modules which are using temporary files (for
example v.random is not using).
>
> Can you please try this variable?
> http://grass.osgeo.org/grass70/manuals/variables.html
>
> --> "TMPDIR, TEMP, TMP" (name depends on the operating system)

Thanks for this tip, I will try.

To be sure, I mean temporary directory located in mapset:
$MAPSET/.tmp/$HOSTNAME/vector . There is already some variable called
GRASS_VECTOR_TEMPORARY, but according description, it is just enabling and
disabling temporary dir. I will play with both and report back.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:16>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 1, 2015, 8:54pm

#2185: Painfully Slow 'v.in.ogr' Vector Import
------------------------------------------------+---------------------------
Reporter: justinzane | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Keywords: import, OGR, performance, v.in.ogr | Platform: Linux
Cpu: x86-64 |
------------------------------------------------+---------------------------

Comment(by martinl):

Replying to [comment:16 imincik1]:

> To be sure, I mean temporary directory located in mapset:
$MAPSET/.tmp/$HOSTNAME/vector . There is already some variable called
GRASS_VECTOR_TEMPORARY, but according description, it is just enabling and
disabling temporary dir. I will play with both and report back.

yes, if `GRASS_VECTOR_TEMPORARY` is defined than a vector map is created
in `$MAPSET/.tmp/$HOSTNAME/vector`.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:17>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 1, 2015, 11:03pm

#2185: Painfully Slow 'v.in.ogr' Vector Import
------------------------------------------------+---------------------------
Reporter: justinzane | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Keywords: import, OGR, performance, v.in.ogr | Platform: Linux
Cpu: x86-64 |
------------------------------------------------+---------------------------

Comment(by imincik1):

TMPDIR, TEMP, TMP has no effect on temporary directory path created by
vector library.

I did one more test on importing some vector data by v.in.ogr

1. temporary directory located on NFS = 16 seconds
2. temporary directory located on NBD = 2 seconds
3. temporary directory located in memory (tmpfs on /mnt/booster type
tmpfs): 0.8 seconds !

It seems that possibility to set temporary directory path created by
vector library to some quicker file system can be beneficial for all
users.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:18>
GRASS GIS <http://grass.osgeo.org>

GDAL · February 2, 2015, 12:00am

#2185: Painfully Slow 'v.in.ogr' Vector Import
------------------------------------------------+---------------------------
Reporter: justinzane | Owner: grass-dev@…
Type: defect | Status: new
Priority: normal | Milestone: 7.0.0
Component: Vector | Version: svn-trunk
Keywords: import, OGR, performance, v.in.ogr | Platform: Linux
Cpu: x86-64 |
------------------------------------------------+---------------------------

Comment(by wenzeslaus):

Replying to [comment:15 neteler]:
> Replying to [comment:14 imincik1]:
> ...
> > 2. What was a great change was moving '.tmp' directory from my mapset
outside of NFS (by using symlink). v.in.ogr was 8 times faster.
> > So my idea is, is it possible to have some environment variable, which
could define global alternative directory for temporary files ? This could
improve situation for all modules which are using temporary files (for
example v.random is not using).
>
> Can you please try this variable?
> http://grass.osgeo.org/grass70/manuals/variables.html
>
> --> "TMPDIR, TEMP, TMP" (name depends on the operating system)

Manual is not completely clear about what is driven by those but is says
that `/tmp` (or perhaps its platform-dependent equivalent, i.e. system tmp
dir) is the default which suggests that it is a different directory than
`/path/to/mapset/.tmp` (i.e. GRASS-specific dir).

{{{
TMPDIR, TEMP, TMP
     [Various GRASS GIS commands and wxGUI]
     The default wxGUI temporary directory is chosen from a platform-
dependent list,
     but the user can control the selection of this directory by setting
one of the
     TMPDIR, TEMP or TMP environment variables Hence the wxGUI uses $TMPDIR
if it
     is set, then $TEMP, otherwise /tmp.
}}}

I'm not sure what about the other things but for wxGUI it seems that if
your `grassdata` dir is at NFS it might be more advantageous if the tmp
dir for GUI would the the system one because it would/could work much
faster (supposing the case of generating images for map display). On the
other hand if you are creating something really big, e.g. animation with
g.gui.animation it might be better if you use the bigger space (which
might be the .tmp dir in `grassdata` at NFS).

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2185#comment:19>
GRASS GIS <http://grass.osgeo.org>