[GRASS-user] The upper limit for displaying large vector layers

GRASS GRASS user
,
1

Hi,
the current GRASS versions share a constraint regarding the maximum size of _displayable_ vector layers.
This affects both traditional GRASS monitors and map displays.

While GRASS is capable to ingest and process rather large vector data sets, these can _not_ be properly be displayed when their size exceeds a certain threshold [which needs to be further pinpointed].

The effect can be experienced when using the GADM dataset. It is provided here:
http://biogeo.berkeley.edu/gadm/data/gadm_v0dot9_shp.zip

The shape file holds about 600+Mb of areas (global administration boundaries).
While _working_ with the data poses no problem (just as expected), attempts to display (d.vect... ) it in GRASS have failed so far.

It is interesting that visualizing data of this size freezes GRASS and QGIS, yet is no problem for Java-based applications like uDig once the heapsize is cranked up.

If other people also encountered this issue, we might be able to better describe this (current) limitation and to tackle it in a next step: Todays "very large data sets" will be the standard data sets of the future.

Peter

--
Dr. Peter Löwe
<peter.loewe@gmx.de>

Sensationsangebot verlängert: GMX FreeDSL - Telefonanschluss + DSL
für nur 16,37 Euro/mtl.!* http://dsl.gmx.de/?ac=OM.AD.PD003K1308T4569a

2

hi Peter,

On Wed, Dec 10, 2008 at 9:09 AM, <peter.loewe@gmx.de> wrote:

Hi,
the current GRASS versions share a constraint regarding the maximum
size of _displayable_ vector layers.
This affects both traditional GRASS monitors and map displays.

[which GRASS version, which OS?]

While GRASS is capable to ingest and process rather large vector data sets,
these can _not_ be properly be displayed when their size exceeds a certain
threshold [which needs to be further pinpointed].

What does really happen? Crash? Slow? Missing vectors?
If crash (maybe the best case), please run it in gdb/ddd and
create a backtrace (see GRASS Debuging in the Wiki). Then
we could try to find the bottleneck (maybe integer overflow).

The effect can be experienced when using the GADM dataset. It is provided here:
http://biogeo.berkeley.edu/gadm/data/gadm_v0dot9_shp.zip

The shape file holds about 600+Mb of areas (global administration boundaries).

You can import it successfully with v.in.ogr?

If successful, could you make the GRASS *location* available?
Would save us time to avoid the (lengthy?) v.in.ogr step.

While _working_ with the data poses no problem (just as expected),
attempts to display (d.vect... ) it in GRASS have failed so far.

In which sense?

It is interesting that visualizing data of this size freezes GRASS and QGIS,
yet is no problem for Java-based applications like uDig once the heapsize is cranked up.

Sure because uDig is non-topological. That's way easier to handle.

Please send a precise description of the problem.

Markus

3

Maybe related:

http://trac.osgeo.org/grass/ticket/384
http://trac.osgeo.org/qgis/ticket/1430

Maciek

4

Maciej Sieczka pisze:

Maybe related:

http://trac.osgeo.org/grass/ticket/384
http://trac.osgeo.org/qgis/ticket/1430

And another one - maybe:

https://trac.osgeo.org/qgis/ticket/1427

Maciek

--
Maciej Sieczka
www.sieczka.org

5

On 13/12/08 13:01, Maciej Sieczka wrote:

Maciej Sieczka pisze:

Maybe related:

http://trac.osgeo.org/grass/ticket/384
http://trac.osgeo.org/qgis/ticket/1430

And another one - maybe:

https://trac.osgeo.org/qgis/ticket/1427

See [1] for a previous debate about the issue is.

Glynn, is this in any way addressed in your reforms of the display system ?

In trunk, displaying the same dataset used in the example in [2] (20000 area), I now seem to see longer display times than before:

GRASS 7.0.svn (BELGIQUE):~ > time d.vect ssbel01 fcol=none type=area
real 0m4.815s
user 0m4.292s
sys 0m0.508s

Times before seemed to have been around max. 3 seconds.

Moritz

[1]http://lists.osgeo.org/pipermail/grass-dev/2006-September/025864.html (and previous)
[2]http://lists.osgeo.org/pipermail/grass-dev/2006-August/025269.html

6

Moritz Lennert wrote:

>> Maybe related:
>>
>> http://trac.osgeo.org/grass/ticket/384
>> http://trac.osgeo.org/qgis/ticket/1430
>
> And another one - maybe:
>
> https://trac.osgeo.org/qgis/ticket/1427

See [1] for a previous debate about the issue is.

Glynn, is this in any way addressed in your reforms of the display system ?

Yes. 7.0 uses floating-point throughout, so the half-pixel shifts
caused by integer coordinates are no longer an issue.

In trunk, displaying the same dataset used in the example in [2] (20000
area), I now seem to see longer display times than before:

GRASS 7.0.svn (BELGIQUE):~ > time d.vect ssbel01 fcol=none type=area
real 0m4.815s
user 0m4.292s
sys 0m0.508s

Times before seemed to have been around max. 3 seconds.

Timing comparisons between 6.x and 7.x are apples-to-oranges, as so
much has changed.

If built with --with-cairo, the default display driver is cairo. You
can force the use of the PNG driver (while it still exists) with
GRASS_RENDER_IMMEDIATE=PNG.

Cairo seems to be slower than the PNG driver for simple test cases
(which seems odd, as the PNG driver's polygon filler is quite
inefficient). AFAICT, cairo's primary target is X via the Render
extension. Other targets seem to be functional rather than optimal
(its PostScript output is far from optimal, so I intend to retain the
separate PS driver for the time being).

Also, the display library and d.vect no longer support the various
render= options; there is no clipping or culling of the data before
passing it to the driver. The two main reasons for those features
(overflowing the 16-bit coordinate range used by X, and the overhead
of passing large amounts of data to a separate monitor process) no
longer apply.

With an in-process driver, I wouldn't expect there to be any benefit
to a separate clipping/culling step, as cairo (etc) has to do this
itself anyhow. But it might be doing it in a sub-optimal manner (i.e.
rendering everything then clipping at the raster level).

Also, the GUI should eventually be changed to offload most of the work
to the video hardware (GRASS_CAIRO_DRAWABLE=) rather than doing
everything client-side.

Ultimately, I don't want to waste time optimising code which could
change substantially (also, I don't want to have to modify code which
has been complicated by optimisation).

--
Glynn Clements <glynn@gclements.plus.com>

7

On 16/12/08 16:56, Glynn Clements wrote:

If built with --with-cairo, the default display driver is cairo. You
can force the use of the PNG driver (while it still exists) with
GRASS_RENDER_IMMEDIATE=PNG.

I don't see any significant differences between the two...

Ultimately, I don't want to waste time optimising code which could
change substantially (also, I don't want to have to modify code which
has been complicated by optimisation).

I understand that. I do think, however, that there is a problem with very long display times for large vector maps and that if we could find a solution this would be very welcome.

Moritz