KML WMS GetMap is performing a heavy database load query

Hi,

Sorry in advance, for the extensive POST, but we don’t found an easiest way to explain it!

Scenario:
Using: geoserver 2.25.2 with geotools 31.2
Database: Oracle 19c
Java: 11

Problem:
When exporting KML through WMS service, we’re facing heavy queries on oracle database, because geoserver is not considering the bbox sent in the request.

Example:
…/geoserver/OSP/wms?service=WMS&version=1.1.0&request=GetMap&layers=OSP%3ALINES&BBOX=-963904.28267229%2C4955818.584451%2C-963678.25599103%2C4955934.285031&WIDTH=1514&HEIGHT=775&srs=EPSG%3A3857&styles=&format=application%2Fvnd.google-earth.kml

Resulting query to obtain a “lookAt” of KML inside “Folder” tag:

SELECT SDO_AGGR_MBR(GEOM), SDO_AGGR_MBR(BOX_GEOM), SDO_AGGR_MBR(LABEL_GEOM)
  FROM MYSCHEMA.V_GS_LINES

Resulting problem:
As you can see, this query is going to fetch the entire table referenced on V_GS_LINES, to perform the SDO_AGGR_MBR for all the geometry columns. This is super overkill, because we’ve millions of records on this table.


KML output:
Attached KML of this example: MYSCHEMA-LINES.kml (8.1 KB)

We’re not experts on KML :slight_smile: , but we’re expecting the following on the output:

  • The “LookAt” of “Placemark” must represent the bbox of the feature
  • The “LookAt” of “Folder” must represent the bbox of the “Placemark”
  • The “LookAt” of “Document” must represent the bbox of the “Folder”

But, what we see when exporting is the following:

  • The “LookAt” of “Placemark” is - OK
  • The “LookAt” of “Folder” is pointing to the center of Portugal (in our case, probably because the query generated, is getting the MBR of all the V_GS_LINES features) - NOK
  • The “LookAt” of “Document” seems to be pointing to the original request bbox, not showing the entire features. (for us this could be the behavior) - NOK

KML structure example:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<kml ....>
	<Document>
		<LookAt>
			<longitude>-8.657884285410205</longitude>
			<latitude>40.6160733058264</latitude>
			<altitude>238.26560720816633</altitude>
			<heading>0.0</heading>
			<tilt>0.0</tilt>
			<range>192.53110805327475</range>
			<altitudeMode>clampToGround</altitudeMode>
		</LookAt>
		<Folder>
			<name>MYSCHEMA:LINES</name>
			<LookAt>
				<longitude>-7.940855119558194</longitude>
				<latitude>39.35544189442219</latitude>
				<altitude>694315.7492837217</altitude>
				<heading>0.0</heading>
				<tilt>0.0</tilt>
				<range>561043.543442022</range>
				<altitudeMode>clampToGround</altitudeMode>
			</LookAt>
			<Placemark id="NODES.4542733">
				<name>ARMVerdemilho &gt;&gt; PTEsteiroMedela (192,64m)</name>
				<LookAt>
					<longitude>-8.657990952662047</longitude>
					<latitude>40.61705045026257</latitude>
					<altitude>204.95134092055306</altitude>
					<heading>0.0</heading>
					<tilt>0.0</tilt>
					<range>165.61143350397117</range>
					<altitudeMode>clampToGround</altitudeMode>
				</LookAt>
				.....
			</Placemark>
		</Folder>
	</Document>
</kml>

Our code analysis:
When exporting, the Geoserver calls the “LookAtDecoratorFactory” class, to get the “LookAt” of the “Folder” tag:

class LayerLookAtDecorator implements KmlDecorator {

        @Override
        public Feature decorate(Feature feature, KmlEncodingContext context) {
 ----->     Envelope bounds = context.getCurrentLayer().getBounds();
            LookAt lookAt = buildLookAt(bounds, context.getLookAtOptions(), false);
            feature.setAbstractView(lookAt);

            return feature;
        }
    }

Reaching Geotools class “JDBCDataStore”, to build the bbox for the multiple geometries:

void buildEnvelopeAggregates(SimpleFeatureType featureType, StringBuffer sql) {
     // walk through all geometry attributes and build the query
    for (AttributeDescriptor attribute : featureType.getAttributeDescriptors()) {
        if (attribute instanceof GeometryDescriptor) {
            String geometryColumn = attribute.getLocalName();
 ----->     dialect.encodeGeometryEnvelope(featureType.getTypeName(), geometryColumn, sql);
            sql.append(",");
        }
    }
    sql.setLength(sql.length() - 1);
}

Calling the “OracleDialect” class that calculates the “SDO_AGGR_MBR”:

    @Override
    public void encodeGeometryEnvelope(String tableName, String geometryColumn, StringBuffer sql) {
        sql.append("SDO_AGGR_MBR(");
        encodeColumnName(null, geometryColumn, sql);
        sql.append(")");
    }

It calculates the bounds, according to the geoserver layer bounds (discarding original bbox), instead of only calculate the bounds of the features that the KML layer has.

So, our question is:

  • Why the order of generating the “LookAt” tags while building KML are not the following?
  1. Getting placemarks
  2. Getting “Folder/Layer” bbox based on the mbr of all placemarks inside
  3. Getting “Document” bbox based on the mbr of all folders inside
  4. In this way, point 2 and 3 can be done on server-side instead of database-side

KML Documentation:

On Tue, Nov 12, 2024 at 6:40 PM Tiago V via OSGeo Discourse <noreply@discourse.osgeo.org> wrote:

So, our question is:

  • Why the order of generating the “LookAt” tags while building KML are not the following?
  1. Getting placemarks
  2. Getting “Folder/Layer” bbox based on the mbr of all placemarks inside
  3. Getting “Document” bbox based on the mbr of all folders inside
  4. In this way, point 2 and 3 can be done on server-side instead of database-side

Because GeoServer never keeps the whole response in memory to avoid OOM
on large requests, all text based output formats are “streaming” out, reading the bits
from the database, generating a bit of output, throwing away the features, moving on
to the next batch and so on.
So in all formats queries need to be performed in “document order”, and the bbox for the
root Document happens to be first.

Normally that’s not an issue, because in the most commonly used spatial database getting
a bbox aggregate is fast (it’s reading from the spatial index directly), but unfortunately
not in the Oracle case.

Is the query you’re getting exactly the one you reported, without any spatial filter?
If so, the code could be optimized by computing the bbox not on the whole layer, but
giving it at least the bbox in the request as a spatial filter.

Cheers
Andrea

Understood the avoiding all-in-memory part :slight_smile:

Yes. It discards the bbox sent in the request.

You’ve tested on our side, with the following change as a workaround:

class LayerLookAtDecorator implements KmlDecorator {

        @Override
        public Feature decorate(Feature feature, KmlEncodingContext context) {
--          Envelope bounds = context.getCurrentLayer().getBounds();
++          Envelope bounds = context.getRequest().getBounds() != null ? context.getRequest().getBounds() : context.getCurrentLayer().getBounds();

            LookAt lookAt = buildLookAt(bounds, context.getLookAtOptions(), false);
            feature.setAbstractView(lookAt);

            return feature;
        }
    }

In this way, the database query to get the layer bbox is not made at all.
It considers the bbox sent on the request.
I don’t know how to force the usage of spatial filter at this point in code.

Cheers

On Wed, Nov 13, 2024 at 3:44 PM Tiago V via OSGeo Discourse <noreply@discourse.osgeo.org> wrote:

You’ve tested on our side, with the following change as a workaround:

class LayerLookAtDecorator implements KmlDecorator {

        @Override
        public Feature decorate(Feature feature, KmlEncodingContext context) {
--          Envelope bounds = context.getCurrentLayer().getBounds();
++          Envelope bounds = context.getRequest().getBounds() != null ? context.getRequest().getBounds() : context.getCurrentLayer().getBounds();

            LookAt lookAt = buildLookAt(bounds, context.getLookAtOptions(), false);
            feature.setAbstractView(lookAt);

            return feature;
        }
    }

In this way, the database query to get the layer bbox is not made at all.
It considers the bbox sent on the request.

This would be a regression, the data returned could be much smaller than the requested bbox, but also
larger (if large geometries happen to cross the requested area): Google Earth would end up pointing an
area that’s either too large, where finding the data is difficult, or one that does not cover all the data,
leaving out relevant bits.

Maybe try context.getCurrentFeatureCollection().getBounds() instead?
It should run a query that’s computing the actual bbox for the features in the KML, with a spatial filter
testing intersection with the requested bbox.

Cheers
Andrea

It works smoothly, executing the query and considering the bbox!!! :slight_smile:

May you proceed with that change?

Thanks

Hi Tiago,
yes you can make a PR. Make sure to include a test in it, or the PR won’t be considered for merge.
The full list of requirements for a PR is here:
https://github.com/geoserver/geoserver/blob/main/.github/PULL_REQUEST_TEMPLATE.md

You will not need to send the CLA for a one liner change, but the rest of the requirements all
apply, since KML is a module in supported land.

Ok.
We will do it next week.

@aaime-geosolutions ,

PR opened:

I’ve done a unit test using the “Basic polygons” shapefile that already exists in the project.

The bbox that i used was the blue rectangle, in order to grab two squares, and have a bigger final bbox.

Cheers