[GRASSLIST:10763] How can I reduce the loading time of GRASS-layers in UMN Mapserver

Dear list,
I created a mapfile for two identical themes: One of them is a shapfile, the
other one is a grass vector dataset. The shapefile contains a *.qix-file
which was created using the program shptree.
I compared the loading time for the two themes using the command "time wget
-O /dev/null "http://path/to/the.map"\. Result: the grass theme took about 12
seconds (on a local machine), shapefile only 2s.
My Question therefore: is there a possibility to speed up grass layers in
the mapserver application (something like shptree does?)? The attribute
table of the grass-theme is a (default) dbf-file.

Declaration of the grass-layer in the mapfile:

# OGR-Layer aus GRASS
#_# START LAYER l8
LAYER NAME "l8"
TYPE polygon
STATUS ON
    CONNECTIONTYPE OGR
    CONNECTION
"/home/wqual/grassdata/muenchen/stadtbezirk/vector/block/head"
    DATA "1"
    METADATA
      LTITLE "l8"
    END #metadata
### TYPE Polygon -- LEGTYPE 0x01 -- SYMTYPE 0x02
### singlepolyleg
   CLASSITEM "cat"
   CLASS NAME "Block_GRASS"
    COLOR 255 0 0
    OUTLINECOLOR 20 200 10
   END #CLASS
#_# LAYER l8 QUERY
END # LAYER l8

I would be very happy to receive some hints/tips on this issue.

Best regards,

Wolfgang

--
Echte DSL-Flatrate dauerhaft für 0,- Euro*!
"Feel free" mit GMX DSL! http://www.gmx.net/de/go/dsl

GRASS does not save spatial index in file becouse it
occupied too much space and spatial index is used only occasionly.
So spatial index could make it slower but 6 times seems too much.

My guess is that it is caused by attributes. Try to go to the
vector directory in GRASS and rename temporarily 'dbln' file.

Also for testing to avoid the difference between nature
of topological and SF format try to test with points or lines.

Radim

On 3/6/06, Wolfgang Qual <Wolfgang.Qual@gmx.net> wrote:

Dear list,
I created a mapfile for two identical themes: One of them is a shapfile, the
other one is a grass vector dataset. The shapefile contains a *.qix-file
which was created using the program shptree.
I compared the loading time for the two themes using the command "time wget
-O /dev/null "http://path/to/the.map&quot;\. Result: the grass theme took about 12
seconds (on a local machine), shapefile only 2s.
My Question therefore: is there a possibility to speed up grass layers in
the mapserver application (something like shptree does?)? The attribute
table of the grass-theme is a (default) dbf-file.

Declaration of the grass-layer in the mapfile:

# OGR-Layer aus GRASS
#_# START LAYER l8
LAYER NAME "l8"
TYPE polygon
STATUS ON
    CONNECTIONTYPE OGR
    CONNECTION
"/home/wqual/grassdata/muenchen/stadtbezirk/vector/block/head"
    DATA "1"
    METADATA
      LTITLE "l8"
    END #metadata
### TYPE Polygon -- LEGTYPE 0x01 -- SYMTYPE 0x02
### singlepolyleg
   CLASSITEM "cat"
   CLASS NAME "Block_GRASS"
    COLOR 255 0 0
    OUTLINECOLOR 20 200 10
   END #CLASS
#_# LAYER l8 QUERY
END # LAYER l8

I would be very happy to receive some hints/tips on this issue.

Best regards,

Wolfgang

--
Echte DSL-Flatrate dauerhaft für 0,- Euro*!
"Feel free" mit GMX DSL! http://www.gmx.net/de/go/dsl

Hi Radim,
thanks for your reply.
Am Dienstag 07 März 2006 pH:44:09 vormittags schrieb Radim Blazek:

GRASS does not save spatial index in file becouse it
occupied too much space and spatial index is used only occasionly.

This means that it is possible to create this spatial index?

So spatial index could make it slower but 6 times seems too much.

My guess is that it is caused by attributes. Try to go to the
vector directory in GRASS and rename temporarily 'dbln' file.

Ok. Loading time decreased significantly. But no dbln, no queries (or am I
wrong?). Mapserver-Maps should be queriable _and_ fast.

Also for testing to avoid the difference between nature
of topological and SF format try to test with points or lines.

I did not test this by now.

Best regards,

Wolfgang

---8<-----------------

Radim

On 3/6/06, Wolfgang Qual <Wolfgang.Qual@gmx.net> wrote:
> Dear list,
> I created a mapfile for two identical themes: One of them is a shapfile,
> the other one is a grass vector dataset. The shapefile contains a
> *.qix-file which was created using the program shptree.
> I compared the loading time for the two themes using the command "time
> wget -O /dev/null "http://path/to/the.map&quot;\. Result: the grass theme took
> about 12 seconds (on a local machine), shapefile only 2s.
> My Question therefore: is there a possibility to speed up grass layers in
> the mapserver application (something like shptree does?)? The attribute
> table of the grass-theme is a (default) dbf-file.
>
> Declaration of the grass-layer in the mapfile:
>
> # OGR-Layer aus GRASS
> #_# START LAYER l8
> LAYER NAME "l8"
> TYPE polygon
> STATUS ON
> CONNECTIONTYPE OGR
> CONNECTION
> "/home/wqual/grassdata/muenchen/stadtbezirk/vector/block/head"
> DATA "1"
> METADATA
> LTITLE "l8"
> END #metadata
> ### TYPE Polygon -- LEGTYPE 0x01 -- SYMTYPE 0x02
> ### singlepolyleg
> CLASSITEM "cat"
> CLASS NAME "Block_GRASS"
> COLOR 255 0 0
> OUTLINECOLOR 20 200 10
> END #CLASS
> #_# LAYER l8 QUERY
> END # LAYER l8
>
> I would be very happy to receive some hints/tips on this issue.
>
> Best regards,
>
> Wolfgang
>
> --
> Echte DSL-Flatrate dauerhaft für 0,- Euro*!
> "Feel free" mit GMX DSL! http://www.gmx.net/de/go/dsl

On 3/7/06, wqual <wolfgang.qual@gmx.net> wrote:

Hi Radim,
thanks for your reply.
Am Dienstag 07 März 2006 pH:44:09 vormittags schrieb Radim Blazek:
> GRASS does not save spatial index in file becouse it
> occupied too much space and spatial index is used only occasionly.

This means that it is possible to create this spatial index?

Spatial index is created on-the-fly if necessary.

> So spatial index could make it slower but 6 times seems too much.
>
> My guess is that it is caused by attributes. Try to go to the
> vector directory in GRASS and rename temporarily 'dbln' file.

Ok. Loading time decreased significantly. But no dbln, no queries (or am I
wrong?). Mapserver-Maps should be queriable _and_ fast.

DBF driver in GRASS creates copy of the table in memory.
Try to use Postgres and SQLite. Postgres library also loads
all results to memory by default.

Unfortunattely GetFeature in OGR reads both geometry and
attributes even if the attributes are not required by application.

Try also to compare time for GRASS and Shapefile if you use attributes
query.

BTW: Attribute queries are passed to GRASS database backend.

Radim

> Also for testing to avoid the difference between nature
> of topological and SF format try to test with points or lines.
I did not test this by now.

Best regards,

Wolfgang

---8<-----------------
>
> Radim
>
> On 3/6/06, Wolfgang Qual <Wolfgang.Qual@gmx.net> wrote:
> > Dear list,
> > I created a mapfile for two identical themes: One of them is a shapfile,
> > the other one is a grass vector dataset. The shapefile contains a
> > *.qix-file which was created using the program shptree.
> > I compared the loading time for the two themes using the command "time
> > wget -O /dev/null "http://path/to/the.map&quot;\. Result: the grass theme took
> > about 12 seconds (on a local machine), shapefile only 2s.
> > My Question therefore: is there a possibility to speed up grass layers in
> > the mapserver application (something like shptree does?)? The attribute
> > table of the grass-theme is a (default) dbf-file.
> >
> > Declaration of the grass-layer in the mapfile:
> >
> > # OGR-Layer aus GRASS
> > #_# START LAYER l8
> > LAYER NAME "l8"
> > TYPE polygon
> > STATUS ON
> > CONNECTIONTYPE OGR
> > CONNECTION
> > "/home/wqual/grassdata/muenchen/stadtbezirk/vector/block/head"
> > DATA "1"
> > METADATA
> > LTITLE "l8"
> > END #metadata
> > ### TYPE Polygon -- LEGTYPE 0x01 -- SYMTYPE 0x02
> > ### singlepolyleg
> > CLASSITEM "cat"
> > CLASS NAME "Block_GRASS"
> > COLOR 255 0 0
> > OUTLINECOLOR 20 200 10
> > END #CLASS
> > #_# LAYER l8 QUERY
> > END # LAYER l8
> >
> > I would be very happy to receive some hints/tips on this issue.
> >
> > Best regards,
> >
> > Wolfgang
> >
> > --
> > Echte DSL-Flatrate dauerhaft für 0,- Euro*!
> > "Feel free" mit GMX DSL! http://www.gmx.net/de/go/dsl

Hello Radim, list.

> This means that it is possible to create this spatial index?

Spatial index is created on-the-fly if necessary.

Sorry for asking, but what means "if necessary"?

> > So spatial index could make it slower but 6 times seems too much.
> >
> > My guess is that it is caused by attributes. Try to go to the
> > vector directory in GRASS and rename temporarily 'dbln' file.
>
> Ok. Loading time decreased significantly. But no dbln, no queries (or am
> I wrong?). Mapserver-Maps should be queriable _and_ fast.

DBF driver in GRASS creates copy of the table in memory.
Try to use Postgres and SQLite. Postgres library also loads
all results to memory by default.

I did not check Postgres, but using SQLite did not "produce" better results.
Took about 12-13s to load the map.

Unfortunattely GetFeature in OGR reads both geometry and
attributes even if the attributes are not required by application.

Try also to compare time for GRASS and Shapefile if you use attributes
query.

Indexed shapefile is still very fast, even when performing queries.

BTW: Attribute queries are passed to GRASS database backend.

And this means that ...?
Do you think, it could be helpful to contact the UMN Mapserver list regarding
this issue?

Cheers,

Wolfgang

---8<-----------------------

Radim

> > Also for testing to avoid the difference between nature
> > of topological and SF format try to test with points or lines.
>
> I did not test this by now.
>
> Best regards,
>
> Wolfgang
>
> ---8<-----------------
>
> > Radim
> >
> > On 3/6/06, Wolfgang Qual <Wolfgang.Qual@gmx.net> wrote:
> > > Dear list,
> > > I created a mapfile for two identical themes: One of them is a
> > > shapfile, the other one is a grass vector dataset. The shapefile
> > > contains a *.qix-file which was created using the program shptree.
> > > I compared the loading time for the two themes using the command
> > > "time wget -O /dev/null "http://path/to/the.map&quot;\. Result: the grass
> > > theme took about 12 seconds (on a local machine), shapefile only 2s.
> > > My Question therefore: is there a possibility to speed up grass
> > > layers in the mapserver application (something like shptree does?)?
> > > The attribute table of the grass-theme is a (default) dbf-file.
> > >
> > > Declaration of the grass-layer in the mapfile:
> > >
> > > # OGR-Layer aus GRASS
> > > #_# START LAYER l8
> > > LAYER NAME "l8"
> > > TYPE polygon
> > > STATUS ON
> > > CONNECTIONTYPE OGR
> > > CONNECTION
> > > "/home/wqual/grassdata/muenchen/stadtbezirk/vector/block/head"
> > > DATA "1"
> > > METADATA
> > > LTITLE "l8"
> > > END #metadata
> > > ### TYPE Polygon -- LEGTYPE 0x01 -- SYMTYPE 0x02
> > > ### singlepolyleg
> > > CLASSITEM "cat"
> > > CLASS NAME "Block_GRASS"
> > > COLOR 255 0 0
> > > OUTLINECOLOR 20 200 10
> > > END #CLASS
> > > #_# LAYER l8 QUERY
> > > END # LAYER l8
> > >
> > > I would be very happy to receive some hints/tips on this issue.
> > >
> > > Best regards,
> > >
> > > Wolfgang
> > >
> > > --
> > > Echte DSL-Flatrate dauerhaft für 0,- Euro*!
> > > "Feel free" mit GMX DSL! http://www.gmx.net/de/go/dsl

On 3/7/06, wqual <wolfgang.qual@gmx.net> wrote:

> DBF driver in GRASS creates copy of the table in memory.
> Try to use Postgres and SQLite. Postgres library also loads
> all results to memory by default.
I did not check Postgres, but using SQLite did not "produce" better results.
Took about 12-13s to load the map.

Then I dont know you have to inspect better whole chain dbdriver-grass-ogr.

Radim

> Unfortunattely GetFeature in OGR reads both geometry and
> attributes even if the attributes are not required by application.
>
> Try also to compare time for GRASS and Shapefile if you use attributes
> query.
Indexed shapefile is still very fast, even when performing queries.
>
> BTW: Attribute queries are passed to GRASS database backend.
And this means that ...?
Do you think, it could be helpful to contact the UMN Mapserver list regarding
this issue?

Cheers,

Wolfgang

---8<-----------------------
>
> Radim
>
> > > Also for testing to avoid the difference between nature
> > > of topological and SF format try to test with points or lines.
> >
> > I did not test this by now.
> >
> > Best regards,
> >
> > Wolfgang
> >
> > ---8<-----------------
> >
> > > Radim
> > >
> > > On 3/6/06, Wolfgang Qual <Wolfgang.Qual@gmx.net> wrote:
> > > > Dear list,
> > > > I created a mapfile for two identical themes: One of them is a
> > > > shapfile, the other one is a grass vector dataset. The shapefile
> > > > contains a *.qix-file which was created using the program shptree.
> > > > I compared the loading time for the two themes using the command
> > > > "time wget -O /dev/null "http://path/to/the.map&quot;\. Result: the grass
> > > > theme took about 12 seconds (on a local machine), shapefile only 2s.
> > > > My Question therefore: is there a possibility to speed up grass
> > > > layers in the mapserver application (something like shptree does?)?
> > > > The attribute table of the grass-theme is a (default) dbf-file.
> > > >
> > > > Declaration of the grass-layer in the mapfile:
> > > >
> > > > # OGR-Layer aus GRASS
> > > > #_# START LAYER l8
> > > > LAYER NAME "l8"
> > > > TYPE polygon
> > > > STATUS ON
> > > > CONNECTIONTYPE OGR
> > > > CONNECTION
> > > > "/home/wqual/grassdata/muenchen/stadtbezirk/vector/block/head"
> > > > DATA "1"
> > > > METADATA
> > > > LTITLE "l8"
> > > > END #metadata
> > > > ### TYPE Polygon -- LEGTYPE 0x01 -- SYMTYPE 0x02
> > > > ### singlepolyleg
> > > > CLASSITEM "cat"
> > > > CLASS NAME "Block_GRASS"
> > > > COLOR 255 0 0
> > > > OUTLINECOLOR 20 200 10
> > > > END #CLASS
> > > > #_# LAYER l8 QUERY
> > > > END # LAYER l8
> > > >
> > > > I would be very happy to receive some hints/tips on this issue.
> > > >
> > > > Best regards,
> > > >
> > > > Wolfgang
> > > >
> > > > --
> > > > Echte DSL-Flatrate dauerhaft für 0,- Euro*!
> > > > "Feel free" mit GMX DSL! http://www.gmx.net/de/go/dsl