[GRASS-dev] t.list slow

GRASS GIS GRASS GIS developer
1

Hi,
I found t.list module to be quite slow, about 500ms. Simple "select *
from strds_absolute_time;" takes about 8ms with sqlite3 or db.select.
If I want to get list of more types for more mapsets (for example
mapsets which are not in SEARCH_PATH) it takes seconds.

Is there a chance to make it faster?

Radim

2

Hi Radim,
the reason why t.list and many other t.* modules are "slow" is the
initialization of the temporal framework. Several background processes
are started and the temporal databases are checked for each mapset in
the mapset searchpath.

You have to use the GRASS GIS temporal framework directly to make it
faster, so that the temporal framework is only initiated once and not
every time a t.* module is called.
I have implemented such a functionality here:
https://bitbucket.org/huhabla/grass-data-explorer/src/03e2fc5b74da163a2da5bfedca664afb542c4077/grassdataexplorerdialog.py?at=master&fileviewer=file-view-default#grassdataexplorerdialog.py-739

The function from the temporal framework is: tgis.get_dataset_list()
This should be almost as fast as the direct sqlite interface.

Best regards
Soeren

2015-10-18 13:15 GMT+02:00 Radim Blazek <radim.blazek@gmail.com>:

Hi,
I found t.list module to be quite slow, about 500ms. Simple "select *
from strds_absolute_time;" takes about 8ms with sqlite3 or db.select.
If I want to get list of more types for more mapsets (for example
mapsets which are not in SEARCH_PATH) it takes seconds.

Is there a chance to make it faster?

Radim

3

On Sun, Oct 18, 2015 at 1:42 PM, Sören Gebbert
<soerengebbert@googlemail.com> wrote:

Hi Radim,
the reason why t.list and many other t.* modules are "slow" is the
initialization of the temporal framework. Several background processes
are started and the temporal databases are checked for each mapset in
the mapset searchpath.

The time is almost constant regardless how many mapsets are in
SEARCH_PATH and still about 500ms if SEARCH_PATH is empty so it must
be something else.

You have to use the GRASS GIS temporal framework directly to make it
faster, so that the temporal framework is only initiated once and not
every time a t.* module is called.

I would prefer to use it directly but I need a C API. Could be some
basic functions like get_dataset_list() ported to C library? If not,
is it some trick, some environment variable or something like that to
avoid unnecessary initialization, which could make it faster?

Radim

I have implemented such a functionality here:
https://bitbucket.org/huhabla/grass-data-explorer/src/03e2fc5b74da163a2da5bfedca664afb542c4077/grassdataexplorerdialog.py?at=master&fileviewer=file-view-default#grassdataexplorerdialog.py-739

The function from the temporal framework is: tgis.get_dataset_list()
This should be almost as fast as the direct sqlite interface.

Best regards
Soeren

2015-10-18 13:15 GMT+02:00 Radim Blazek <radim.blazek@gmail.com>:

Hi,
I found t.list module to be quite slow, about 500ms. Simple "select *
from strds_absolute_time;" takes about 8ms with sqlite3 or db.select.
If I want to get list of more types for more mapsets (for example
mapsets which are not in SEARCH_PATH) it takes seconds.

Is there a chance to make it faster?

Radim

4

Hi Radim,

2015-10-18 14:08 GMT+02:00 Radim Blazek <radim.blazek@gmail.com>:

On Sun, Oct 18, 2015 at 1:42 PM, Sören Gebbert
<soerengebbert@googlemail.com> wrote:

Hi Radim,
the reason why t.list and many other t.* modules are "slow" is the
initialization of the temporal framework. Several background processes
are started and the temporal databases are checked for each mapset in
the mapset searchpath.

The time is almost constant regardless how many mapsets are in
SEARCH_PATH and still about 500ms if SEARCH_PATH is empty so it must
be something else.

My Python profiler shows me that the most time is spend by calling the
C-library access subprocess, some GRASS modules and in the temporal
database check.

You have to use the GRASS GIS temporal framework directly to make it
faster, so that the temporal framework is only initiated once and not
every time a t.* module is called.

I would prefer to use it directly but I need a C API. Could be some
basic functions like get_dataset_list() ported to C library? If not,
is it some trick, some environment variable or something like that to
avoid unnecessary initialization, which could make it faster?

Unfortunately not. The initialization is important and cant be
skipped, since a lot of functionality depends on it.
Almost 95% of the GRASS GIS temporal framework is implemented in
Python, hence all of the functionality.
There is no C/C++ API available to access the Python implementation.

A work around would be to implement a Python script that calls all the
Python API parts to generate the desired text output, so substitute
several t.list and t.rast.list calls.

Best regards
Soeren

Radim

I have implemented such a functionality here:
https://bitbucket.org/huhabla/grass-data-explorer/src/03e2fc5b74da163a2da5bfedca664afb542c4077/grassdataexplorerdialog.py?at=master&fileviewer=file-view-default#grassdataexplorerdialog.py-739

The function from the temporal framework is: tgis.get_dataset_list()
This should be almost as fast as the direct sqlite interface.

Best regards
Soeren

2015-10-18 13:15 GMT+02:00 Radim Blazek <radim.blazek@gmail.com>:

Hi,
I found t.list module to be quite slow, about 500ms. Simple "select *
from strds_absolute_time;" takes about 8ms with sqlite3 or db.select.
If I want to get list of more types for more mapsets (for example
mapsets which are not in SEARCH_PATH) it takes seconds.

Is there a chance to make it faster?

Radim