[Geoserver-users] GML QName issues with PostGIS feature store

GeoServer GeoServer User
,
1

Hello everyone,

We are running a Geoserver instance with Postgres/PostGIS backend. Everything
seems to work fine so far, except for WFS with GML.

We have two specific issues here:

1) The auto-generated feature IDs (see [1]) do not comply with the QName
convention. We use UUIDs to identify our datasets and Geoserver seems to uses
this UUID directly to generate Feature IDs. Since QNames must not start with a
number, we get invalid GML for roughly 10 out of 16 datasets. Most
Validators/Clients we have tested fail to parse this GML (QGIS among them).

2) Geoserver uses Postgres column names as feature attribute names. Since
Postgres column names are not restricted to QNames (e.g. "12345", "km/s" are
valid column names), we sometimes run into similar issues here.

Is this intended behavior, i.e. is the dataset provider responsible for
ensuring QName compliant attribute names, or is it a bug?

Kind regards,
Matthias

p.s.: Geoserver version is 2.8.3

[1]:

<gml:featureMember>
  <ckandev:402b28fe-af35-463f-a78f-47f121cea236
    fid="402b28fe-af35-463f-a78f-47f121cea236.1">
    <ckandev:X>51.003874</ckandev:X>
    <ckandev:Y>13.726043</ckandev:Y>
    <ckandev:Timestamp>2012-07-20T:10:00GMT+1</ckandev:Timestamp>
    <ckandev:Place>Bannewitz</ckandev:Place>
    <ckandev:Zn_1000_-_400___micro_g_per_g_>127.3936706</ckandev:Zn_1000_-_400___micro_g_per_g_>
    <ckandev:Zn_400_-_100___micro_g_per_g_>418.9342066</ckandev:Zn_400_-_100___micro_g_per_g_>
    <ckandev:Zn_100_-_63___micro_g_per_g_>743.5718278</ckandev:Zn_100_-_63___micro_g_per_g_>
    <ckandev:Zn_63_-_0.45___micro_g_per_g_>528.2146161</ckandev:Zn_63_-_0.45___micro_g_per_g_>
    <ckandev:Zn_SUMM___micro_g_per_g_>377.8245751</ckandev:Zn_SUMM___micro_g_per_g_>
    <ckandev:Cu_1000_-_400___micro_g_per_g_>48.0749849</ckandev:Cu_1000_-_400___micro_g_per_g_>
    <ckandev:Cu_400_-_100___micro_g_per_g_>112.1762429</ckandev:Cu_400_-_100___micro_g_per_g_>
    <ckandev:Cu_100_-_63___micro_g_per_g_>164.8965903</ckandev:Cu_100_-_63___micro_g_per_g_>
    <ckandev:Cu_63_-_0.45___micro_g_per_g_>151.0506145</ckandev:Cu_63_-_0.45___micro_g_per_g_>
    <ckandev:Cu_SUMM___micro_g_per_g_>105.0860617</ckandev:Cu_SUMM___micro_g_per_g_>
    <ckandev:Cd_1000_-_400___micro_g_per_g_>0.056641806</ckandev:Cd_1000_-_400___micro_g_per_g_>
    <ckandev:Cd_400_-_100___micro_g_per_g_>0.166999107</ckandev:Cd_400_-_100___micro_g_per_g_>
    <ckandev:Cd_100_-_63___micro_g_per_g_>0.264253773</ckandev:Cd_100_-_63___micro_g_per_g_>
    <ckandev:Cd_63_-_0.45___micro_g_per_g_>0.278181578</ckandev:Cd_63_-_0.45___micro_g_per_g_>
    <ckandev:Cd_SUMM___micro_g_per_g_>0.1663264</ckandev:Cd_SUMM___micro_g_per_g_>
    <ckandev:Shape>
      <gml:Point srsName="http://www.opengis.net/gml/srs/epsg.xml#4326&quot;&gt;
        <gml:coordinates xmlns:gml="http://www.opengis.net/gml&quot;
          decimal="." cs="," ts=" ">13.726043,51.003874</gml:coordinates>
      </gml:Point>
    </ckandev:Shape>
  </ckandev:402b28fe-af35-463f-a78f-47f121cea236>
</gml:featureMember>

2

Hi Matthias,
I believe this is intended behavior, it’s not up to the data store code to police attribute names
and ids. E.g., if you were just using this layer for WMS, there would not be any problem,
serving also WFS is your choice, many do, but not everybody.

We have been over this a few times without any good consensus (and more importantly,
without any funding to take action on it), various positions are:

  • Do not allow publishing if the data is not compatible with all services
  • Warn the admin if a layer is not compatible with some service restrictions
  • Perform on the fly mappings to avoid this issue, with different mappings per service

The quickest solution for you is probably to setup a sql view that amends the names and
ids so that they are compliant. This works as long as you don’t have to edit via WFS-T of course.

Cheers
Andrea

···

On Wed, Jun 15, 2016 at 10:58 AM, Matthias Müller <matthias_mueller@anonymised.com> wrote:

Hello everyone,

We are running a Geoserver instance with Postgres/PostGIS backend. Everything
seems to work fine so far, except for WFS with GML.

We have two specific issues here:

  1. The auto-generated feature IDs (see [1]) do not comply with the QName
    convention. We use UUIDs to identify our datasets and Geoserver seems to uses
    this UUID directly to generate Feature IDs. Since QNames must not start with a
    number, we get invalid GML for roughly 10 out of 16 datasets. Most
    Validators/Clients we have tested fail to parse this GML (QGIS among them).

  2. Geoserver uses Postgres column names as feature attribute names. Since
    Postgres column names are not restricted to QNames (e.g. “12345”, “km/s” are
    valid column names), we sometimes run into similar issues here.

Is this intended behavior, i.e. is the dataset provider responsible for
ensuring QName compliant attribute names, or is it a bug?

Kind regards,
Matthias

p.s.: Geoserver version is 2.8.3

[1]:

gml:featureMember
<ckandev:402b28fe-af35-463f-a78f-47f121cea236
fid=“402b28fe-af35-463f-a78f-47f121cea236.1”>
ckandev:X51.003874</ckandev:X>
ckandev:Y13.726043</ckandev:Y>
ckandev:Timestamp2012-07-20T:10:00GMT+1</ckandev:Timestamp>
ckandev:PlaceBannewitz</ckandev:Place>
ckandev:Zn_1000_-_400___micro_g_per_g_127.3936706</ckandev:Zn_1000_-400___micro_g_per_g>
ckandev:Zn_400_-_100___micro_g_per_g_418.9342066</ckandev:Zn_400_-100___micro_g_per_g>
ckandev:Zn_100_-_63___micro_g_per_g_743.5718278</ckandev:Zn_100_-63___micro_g_per_g>
ckandev:Zn_63_-_0.45___micro_g_per_g_528.2146161</ckandev:Zn_63_-0.45___micro_g_per_g>
ckandev:Zn_SUMM___micro_g_per_g_377.8245751</ckandev:Zn_SUMM___micro_g_per_g_>
ckandev:Cu_1000_-_400___micro_g_per_g_48.0749849</ckandev:Cu_1000_-400___micro_g_per_g>
ckandev:Cu_400_-_100___micro_g_per_g_112.1762429</ckandev:Cu_400_-100___micro_g_per_g>
ckandev:Cu_100_-_63___micro_g_per_g_164.8965903</ckandev:Cu_100_-63___micro_g_per_g>
ckandev:Cu_63_-_0.45___micro_g_per_g_151.0506145</ckandev:Cu_63_-0.45___micro_g_per_g>
ckandev:Cu_SUMM___micro_g_per_g_105.0860617</ckandev:Cu_SUMM___micro_g_per_g_>
ckandev:Cd_1000_-_400___micro_g_per_g_0.056641806</ckandev:Cd_1000_-400___micro_g_per_g>
ckandev:Cd_400_-_100___micro_g_per_g_0.166999107</ckandev:Cd_400_-100___micro_g_per_g>
ckandev:Cd_100_-_63___micro_g_per_g_0.264253773</ckandev:Cd_100_-63___micro_g_per_g>
ckandev:Cd_63_-_0.45___micro_g_per_g_0.278181578</ckandev:Cd_63_-0.45___micro_g_per_g>
ckandev:Cd_SUMM___micro_g_per_g_0.1663264</ckandev:Cd_SUMM___micro_g_per_g_>
ckandev:Shape
<gml:Point srsName=“http://www.opengis.net/gml/srs/epsg.xml#4326”>
<gml:coordinates xmlns:gml=“http://www.opengis.net/gml”
decimal=“.” cs=“,” ts=" ">13.726043,51.003874</gml:coordinates>
</gml:Point>
</ckandev:Shape>
</ckandev:402b28fe-af35-463f-a78f-47f121cea236>
</gml:featureMember>

What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports. http://pubads.g.doubleclick.net/gampad/clk?id=1444514421&iu=/41014381

Geoserver-users mailing list
Geoserver-users@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

3

Hi Andrea,

thanks for the quick reply.

Just for the record: Correct behaviour towards the client also matters in option (2). Possibilities for handling wrong GML are:

2a) return an HTTP error (fail early)
2b) hide invalid layers in Capabilities
2c) disable GML output on a particular WFS endpoint if one of the layers is invalid (contradicts WFS spec!)
2d) disable WFS endpoint until data becomes valid

All of these are far from ideal, so option (2) might be the least favorable solution.

Kind regards,
Matthias

···

Am 15.06.2016 um 11:14 schrieb Andrea Aime:

Hi Matthias,
I believe this is intended behavior, it’s not up to the data store code to police attribute names
and ids. E.g., if you were just using this layer for WMS, there would not be any problem,
serving also WFS is your choice, many do, but not everybody.

We have been over this a few times without any good consensus (and more importantly,
without any funding to take action on it), various positions are:

  • Do not allow publishing if the data is not compatible with all services
  • Warn the admin if a layer is not compatible with some service restrictions
  • Perform on the fly mappings to avoid this issue, with different mappings per service

The quickest solution for you is probably to setup a sql view that amends the names and
ids so that they are compliant. This works as long as you don’t have to edit via WFS-T of course.

Cheers
Andrea

On Wed, Jun 15, 2016 at 10:58 AM, Matthias Müller <matthias_mueller@anonymised.com> wrote:

Hello everyone,

We are running a Geoserver instance with Postgres/PostGIS backend. Everything
seems to work fine so far, except for WFS with GML.

We have two specific issues here:

  1. The auto-generated feature IDs (see [1]) do not comply with the QName
    convention. We use UUIDs to identify our datasets and Geoserver seems to uses
    this UUID directly to generate Feature IDs. Since QNames must not start with a
    number, we get invalid GML for roughly 10 out of 16 datasets. Most
    Validators/Clients we have tested fail to parse this GML (QGIS among them).

  2. Geoserver uses Postgres column names as feature attribute names. Since
    Postgres column names are not restricted to QNames (e.g. “12345”, “km/s” are
    valid column names), we sometimes run into similar issues here.

Is this intended behavior, i.e. is the dataset provider responsible for
ensuring QName compliant attribute names, or is it a bug?

Kind regards,
Matthias

p.s.: Geoserver version is 2.8.3

[1]:

gml:featureMember
<ckandev:402b28fe-af35-463f-a78f-47f121cea236
fid=“402b28fe-af35-463f-a78f-47f121cea236.1”>
ckandev:X51.003874</ckandev:X>
ckandev:Y13.726043</ckandev:Y>
ckandev:Timestamp2012-07-20T:10:00GMT+1</ckandev:Timestamp>
ckandev:PlaceBannewitz</ckandev:Place>
ckandev:Zn_1000_-_400___micro_g_per_g_127.3936706</ckandev:Zn_1000_-400___micro_g_per_g>
ckandev:Zn_400_-_100___micro_g_per_g_418.9342066</ckandev:Zn_400_-100___micro_g_per_g>
ckandev:Zn_100_-_63___micro_g_per_g_743.5718278</ckandev:Zn_100_-63___micro_g_per_g>
ckandev:Zn_63_-_0.45___micro_g_per_g_528.2146161</ckandev:Zn_63_-0.45___micro_g_per_g>
ckandev:Zn_SUMM___micro_g_per_g_377.8245751</ckandev:Zn_SUMM___micro_g_per_g_>
ckandev:Cu_1000_-_400___micro_g_per_g_48.0749849</ckandev:Cu_1000_-400___micro_g_per_g>
ckandev:Cu_400_-_100___micro_g_per_g_112.1762429</ckandev:Cu_400_-100___micro_g_per_g>
ckandev:Cu_100_-_63___micro_g_per_g_164.8965903</ckandev:Cu_100_-63___micro_g_per_g>
ckandev:Cu_63_-_0.45___micro_g_per_g_151.0506145</ckandev:Cu_63_-0.45___micro_g_per_g>
ckandev:Cu_SUMM___micro_g_per_g_105.0860617</ckandev:Cu_SUMM___micro_g_per_g_>
ckandev:Cd_1000_-_400___micro_g_per_g_0.056641806</ckandev:Cd_1000_-400___micro_g_per_g>
ckandev:Cd_400_-_100___micro_g_per_g_0.166999107</ckandev:Cd_400_-100___micro_g_per_g>
ckandev:Cd_100_-_63___micro_g_per_g_0.264253773</ckandev:Cd_100_-63___micro_g_per_g>
ckandev:Cd_63_-_0.45___micro_g_per_g_0.278181578</ckandev:Cd_63_-0.45___micro_g_per_g>
ckandev:Cd_SUMM___micro_g_per_g_0.1663264</ckandev:Cd_SUMM___micro_g_per_g_>
ckandev:Shape
<gml:Point srsName=“http://www.opengis.net/gml/srs/epsg.xml#4326”>
<gml:coordinates xmlns:gml=“http://www.opengis.net/gml”
decimal=“.” cs=“,” ts=" ">13.726043,51.003874</gml:coordinates>
</gml:Point>
</ckandev:Shape>
</ckandev:402b28fe-af35-463f-a78f-47f121cea236>
</gml:featureMember>

What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports. http://pubads.g.doubleclick.net/gampad/clk?id=1444514421&iu=/41014381

Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

4

On Wed, Jun 15, 2016 at 11:58 AM, Matthias Müller <
matthias_mueller@anonymised.com> wrote:

Hi Andrea,

thanks for the quick reply.

Just for the record: Correct behaviour towards the client also matters in
option (2). Possibilities for handling wrong GML are:

2a) return an HTTP error (fail early)
2b) hide invalid layers in Capabilities
2c) disable GML output on a particular WFS endpoint if one of the layers
is invalid (contradicts WFS spec!)
2d) disable WFS endpoint until data becomes valid

All of these are far from ideal, so option (2) might be the least
favorable solution.

You're still a bit too focused, many stand up a GeoServer to only serve a
particular front end application, without
any desire to provide an SDI, and use WFS only to extract geojson out of it.

As said, it's not the job of the software to police a fully valid OGC
service, the software is used in a broader set of
use cases. It would make sense to have an option to be strict in terms of
OGC and in this case probably error out,
or remove the layers that are not compatible.

In any case, the main issue is finding dev resources, the question is not
what's the best option, but what's the one that
can get funding/spare time dev interest enough to be implemented (so far
the answer has been "none").

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------