[Geoserver-users] Odd behaviour with WFS GetFeature in Geoserver 2.7.1.1 - property names (and geometry) returned that are not requested.

If I make this POST query to our server:
http://maps.gns.cri.nz/geoserverlatest (public)

<wfs:GetFeature xmlns:wfs="http://www.opengis.net/wfs&quot; service="WFS" version="1.1.0" count="1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance&quot; xsi:schemaLocation="http://www.opengis.net/wfs http://schemas.opengis.net/wfs/1.1.0/wfs/xsd&quot; outputFormat="application/json"><wfs:Query typeName="gns:PETROLEUM.PETROLEUM_BOREHOLE_VIEW" srsName="EPSG:4326"><wfs:PropertyName>BH_NAME</wfs:PropertyName><ogc:Filter xmlns:ogc="http://www.opengis.net/ogc&quot;&gt;&lt;ogc:Intersects&gt;&lt;ogc:PropertyName&gt;SHAPE&lt;/ogc:PropertyName&gt;&lt;gml:Polygon xmlns:gml="http://www.opengis.net/gml&quot; srsName="EPSG:4326"><gml:exterior><gml:LinearRing><gml:posList>170.0244140625 -44.8714427501659 170.0244140625 -44.68427737181224 169.7607421875 -44.68427737181224 169.7607421875 -44.8714427501659 170.0244140625 -44.8714427501659</gml:posList></gml:LinearRing></gml:exterior></gml:Polygon></ogc:Intersects></ogc:Filter></wfs:Query></wfs:GetFeature>

Then I get three properties returned, BH_NAME, BH_ID, and BH_TYPE_ID (there are a large no. of other properties that could be returned and are if you remove the <wfs:PropertyName>BH_NAME</wfs:PropertyName> from the query).
I expect only BH_NAME.

Worse, (far worse), it returns the geometry despite not having SHAPE in the property lsit. For the above feature type, this doesnt matter but if I make WFS queries on a complex polygon classes to fetch attributes only from a spatial search, then I get swamped by multi-megabytes of geometry in the response.

--
Phil Scadden, Senior Scientist GNS Science Ltd 764 Cumberland St, Private Bag 1930, Dunedin, New Zealand Ph +64 3 4799663, fax +64 3 477 5232

Notice: This email and any attachments are confidential.
If received in error please destroy and immediately notify us.
Do not copy or disclose the contents.

Phil,

your schema makes BH_ID, SHAPE, and BH_TYPE_ID mandatory (minOccurs=1 and nillable=false):
http://maps.gns.cri.nz/geoserverlatest/wfs?service=WFS&version=1.1.0&request=DescribeFeatureType&typeName=gns%3APETROLEUM.PETROLEUM_BOREHOLE_VIEW

I far as I know, JDBC feature sources make a property mandatory by default if the underlying database column has a NOT NULL constraint. You may be able to change this with GeoServer schema mapping:
http://docs.geoserver.org/stable/en/user/services/wfs/schemamapping.html

The GeoJSON spec requires a geometry property but permits it to be null; I do not know whether GeoServer JSON outputFormat encodes a geometry if the geometry property is not mandatory and not requested.

Kind regards,
Ben.

On 15/01/16 11:46, Phil Scadden wrote:

If I make this POST query to our server:
http://maps.gns.cri.nz/geoserverlatest (public)

<wfs:GetFeature xmlns:wfs="http://www.opengis.net/wfs&quot; service="WFS"
version="1.1.0" count="1"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance&quot;
xsi:schemaLocation="http://www.opengis.net/wfs
http://schemas.opengis.net/wfs/1.1.0/wfs/xsd&quot;
outputFormat="application/json"><wfs:Query
typeName="gns:PETROLEUM.PETROLEUM_BOREHOLE_VIEW"
srsName="EPSG:4326"><wfs:PropertyName>BH_NAME</wfs:PropertyName><ogc:Filter
xmlns:ogc="http://www.opengis.net/ogc&quot;&gt;&lt;ogc:Intersects&gt;&lt;ogc:PropertyName&gt;SHAPE&lt;/ogc:PropertyName&gt;&lt;gml:Polygon
xmlns:gml="http://www.opengis.net/gml&quot;
srsName="EPSG:4326"><gml:exterior><gml:LinearRing><gml:posList>170.0244140625
-44.8714427501659 170.0244140625 -44.68427737181224 169.7607421875
-44.68427737181224 169.7607421875 -44.8714427501659 170.0244140625
-44.8714427501659</gml:posList></gml:LinearRing></gml:exterior></gml:Polygon></ogc:Intersects></ogc:Filter></wfs:Query></wfs:GetFeature>

Then I get three properties returned, BH_NAME, BH_ID, and BH_TYPE_ID
(there are a large no. of other properties that could be returned and
are if you remove the <wfs:PropertyName>BH_NAME</wfs:PropertyName> from
the query).
I expect only BH_NAME.

Worse, (far worse), it returns the geometry despite not having SHAPE in
the property lsit. For the above feature type, this doesnt matter but if
I make WFS queries on a complex polygon classes to fetch attributes only
from a spatial search, then I get swamped by multi-megabytes of geometry
in the response.

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

I far as I know, JDBC feature sources make a property mandatory by default if the underlying database column has a NOT NULL constraint. You may be able to change this with GeoServer schema mapping:
http://docs.geoserver.org/stable/en/user/services/wfs/schemamapping.html

That is very interesting. Thanks for that.

The GeoJSON spec requires a geometry property but permits it to be null; I do not know whether GeoServer JSON outputFormat encodes a geometry if the geometry property is not mandatory and not requested.

thanks for that. However, I get geometry even if outputformat is GML or CSV. This didnt happen in earlier geoserver versions.

Notice: This email and any attachments are confidential.
If received in error please destroy and immediately notify us.
Do not copy or disclose the contents.

On 15/01/16 13:48, Phil Scadden wrote:

However, I get geometry even if outputformat is GML or
CSV. This didnt happen in earlier geoserver versions.

What happens if you use schema mapping to make the geometry optional?

The code that looks like it makes a NOT NULL property mandatory seems to have been around for at least five years, but it is possible that WFS is now better at honouring it.

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

On Fri, Jan 15, 2016 at 1:48 AM, Phil Scadden <p.scadden@anonymised.com> wrote:

thanks for that. However, I get geometry even if outputformat is GML or
CSV. This didnt happen in earlier geoserver versions.

I'd be very surprised if that was the case, the engine deciding whether to
include
a property or not in the output does not know which output format is being
used,
it never did as far as I know.
The store level recognition of constraints did not change recently either
(that I know at least)...
how old was the GeoServer you're trying to replace?

Could it be that the table constraints changed around the same time you
upgraded GeoServer instead?
Or have you maybe switched from SDE to direct Oracle spatial usage
(different
store, different behavior).

Thinking out loud, we could have a flag at the store level to make all
properties
nullable even if they are not... and then disallow editing on those same
tables
(since it cannot be done anymore in a predictable fashion, DFT says you can
skip an attribute, but then you cannot).... or just document that at that
point WFS-T
is not going to work reliably.

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

Misunderstanding. I am not asserting that properties returned depend on outFormat. Ben seemed to be suggesting this might be why I am getting geometry. What I am asserting is that earlier version didnt return geometry if it wasnt on the property list.

No. SHAPE is nullable in the underlying SDE store. The maintenance process for the table regularly sets shape to null to force recalculation of the geometry from attributes in the same table. I assert that I am not getting the geometry because shape is set to not null.

I wish… No, still stuck with SDE.

The constraint of requiring not null properties to be returned regardless of whether in property list, is only useful within the context of WFS-T. That should be the flagging criteria. If not in a WFS-T context, then only send properties requested. I suspect that editing is rare but querying is common. I think OGC should consider a nogeometry switch like arcGIS server.

Notice: This email and any attachments are confidential. If received in error please destroy and immediately notify us. Do not copy or disclose the contents.

···

On Fri, Jan 15, 2016 at 1:48 AM, Phil Scadden <p.scadden@anonymised.com> wrote:

thanks for that. However, I get geometry even if outputformat is GML or
CSV. This didnt happen in earlier geoserver versions.

I’d be very surprised if that was the case, the engine deciding whether to include
a property or not in the output does not know which output format is being used,
it never did as far as I know.

Could it be that the table constraints changed around the same time you
upgraded GeoServer instead?

Or have you maybe switched from SDE to direct Oracle spatial usage (different
store, different behavior).

Thinking out loud, we could have a flag at the store level to make all properties
nullable even if they are not… and then disallow editing on those same tables
(since it cannot be done anymore in a predictable fashion, DFT says you can
skip an attribute, but then you cannot)… or just document that at that point WFS-T
is not going to work reliably

On Sun, Jan 17, 2016 at 1:19 PM, Phil Scadden <p.scadden@anonymised.com> wrote:

Thinking out loud, we could have a flag at the store level to make all
properties
nullable even if they are not... and then disallow editing on those same
tables
(since it cannot be done anymore in a predictable fashion, DFT says you can
skip an attribute, but then you cannot).... or just document that at that
point WFS-T
is not going to work reliably

The constraint of requiring not null properties to be returned regardless
of whether in property list, is only useful within the context of WFS-T.
That should be the flagging criteria. If not in a WFS-T context, then only
send properties requested. I suspect that editing is rare but querying is
common. I think OGC should consider a nogeometry switch like arcGIS server.

I agree with you, OGC did not give us a DescribeFeatureType for WFS-T and
one for basic.
DescribeFeatureType generates a XML schema that we have to abide to, at
least for GML outputs (otherwise they would be schema invalid).
Given we have only one, if we make the attributes nullable in the schema
when they are not in the db, we'll be able to ask for the property
to be removed in GetFeature, but the data will not be properly editable. If
we make it editable, then you're also forced to get it in the output...

I agree for "nogeometry", but imho it should be more general, there might
be other fields that are mandatory in the db that you
don't care for in the output. Maybe a "ignoreNonNull=true" property in the
GetFeature request that would force it to just listen
to the properties being requested, instead of also double checking what's
mandatory in the schema (and maybe add the
same param in the DescribeFeatureType backlink that we generate, so that
the output also confirms nothing is mandatory).

Just thinking out loud :-p

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

+1 for a nogeometry switch. We need some way to support GetFeatureInfo but not return geometry, for security reasons. Actually it would be great to support returning a BBOX rather than the actual geometry, since this supports zooming. This improves both security and performance.

···

On Sun, Jan 17, 2016 at 1:19 PM, Phil Scadden <p.scadden@anonymised.com> wrote:

The constraint of requiring not null properties to be returned regardless of whether in property list, is only useful within the context of WFS-T. That should be the flagging criteria. If not in a WFS-T context, then only send properties requested. I suspect that editing is rare but querying is common. I think OGC should consider a nogeometry switch like arcGIS server.