[Geoserver-devel] Stop using a global cached schema for all features types in WFS 1.1.0 for encoding GML3 responses

Hi,

Please check this issue description: https://osgeo-org.atlassian.net/browse/GEOS-8017

So basically the idea is to stop using a global cached schema (for all the feature types) in WFS 1.1.0 but instead build a specific schema for each request using only the necessary feature types similar to what is done in WFS 2.0.0.

Another advantage is that we will also avoid the cost of building a global schema for all the feature types which can take some time when we have a lot of feature types.

I made some performance tests to measure the impact of this solution. I used two GeoServer instances:

  • NORMAL GEOSERVER - 8076 - normal GeoServer that ues a global cached schema

  • CHANGED GEOSERVER - 8075 - changed GeoServer that uses the solution described above

I used three (vector) layers for doing this tests:

  • sf:roads - shipped with GeoServer release
  • topp:roads - shipped with GeoServer release
  • it.geosolutions:cache_test - please find it attached, shapefile with pints that have 30 attributes (integer, string and date)

Each layer features (MAX 3 features per request) were requested using WFS 1.1.0 and WFS 2.0.0 on each GeoServer instance (WFS 2.0.0 results can be used as control values). I run this tests using JMeter (please find the JMeter configuration attached), the results are as follows:

My conclusion is that the proposed solution has an insignificant performance penalty, the throughput is the same for all the cases. I also monitored the two GeoServer instances JVMs using VisualVM:

The CPU usage is a little bit higher in the changed instance and no memory leaks were detected.

Since the above solutions touches how schemas are generated I (try to) run the CITE tests for both instances. Long story short, I was no able to successfully run the tests suites that requires test data to be instantiated (lot’s of errors during the tests). The results are equal on both servers unfortunately with the same tests failed on both:

NORMAL SERVER:

CHANGED SERVER:

A pull request for this was made: https://github.com/geoserver/geoserver/pull/2140

Opinions on this ?

Kind regards,

Nuno Oliveira

(attachments)

cache_test.dbf (3.9 KB)
cache_test.shp (184 Bytes)
cache_test.shx (124 Bytes)
wfs-test.jmx (49.5 KB)

···
-- 
==
GeoServer Professional Services from the experts! 
Visit [http://goo.gl/it488V](http://goo.gl/it488V) for more information.
==
Nuno Miguel Carvalho Oliveira
@nmcoliveira
Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy

phone: +39 0584 962313
fax:   +39 0584 1660272
mob:   +39  333 8128928

[http://www.geo-solutions.it](http://www.geo-solutions.it)
[http://twitter.com/geosolutions_it](http://twitter.com/geosolutions_it)

-------------------------------------------------------

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono
da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate
nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e
-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo
anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.
 
The information in this message and/or attachments, is intended solely for the attention and use of
the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree
June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying,
distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender does not give any warranty or accept liability as the content,
accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which
arise as a result of e-mail transmission, viruses, etc.

On 06/03/17 06:21, Nuno Oliveira wrote:

A pull request for this was made: https://github.com/geoserver/geoserver/pull/2140
Opinions on this ?

Travis CI reports many app-schema unit test failures:
https://s3.amazonaws.com/archive.travis-ci.org/jobs/207942836/log.txt

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

Hi Ben,

Thanks for the warning, pull request amended.

Last minute change and forget to check app-schema tests which proved that my last minute change was actually a bad idea for when schemas are provided by the feature type :frowning:

Kind regards,

Nuno Oliveira

On 03/05/2017 08:10 PM, Ben Caradoc-Davies wrote:

On 06/03/17 06:21, Nuno Oliveira wrote:

A pull request for this was made: https://github.com/geoserver/geoserver/pull/2140
Opinions on this ?

Travis CI reports many app-schema unit test failures:
https://s3.amazonaws.com/archive.travis-ci.org/jobs/207942836/log.txt

Kind regards,

--

GeoServer Professional Services from the experts!
Visit http://goo.gl/it488V for more information.

Nuno Miguel Carvalho Oliveira
@nmcoliveira
Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono
da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate
nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e
-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo
anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.
  The information in this message and/or attachments, is intended solely for the attention and use of
the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree
June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying,
distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender does not give any warranty or accept liability as the content,
accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which
arise as a result of e-mail transmission, viruses, etc.

Hi,

I don't see how to fix this for the use case where app-schema builds a complex feature using feature chaining. The problem is that in the GML3 encoder we only know the root feature type but there is no way (at least I could not found it) to discovered the chained feature types that were used.

So since we cannot find all the feature types that are used we need to always use all the feature types available in the catalog. Note that WFS 2.0.0 has the same issue, using app-schema feature chaining with WFS 2.0.0 will cause some trouble.

So for now, app-schema will still rely on the the full catalog schema.

Any other ides on this ?

Regards,

On 03/06/2017 02:08 PM, Nuno Oliveira wrote:

Hi Ben,

Thanks for the warning, pull request amended.

Last minute change and forget to check app-schema tests which proved
that my last minute change was actually a bad idea for when schemas are
provided by the feature type :frowning:

Kind regards,

Nuno Oliveira

On 03/05/2017 08:10 PM, Ben Caradoc-Davies wrote:

On 06/03/17 06:21, Nuno Oliveira wrote:

A pull request for this was made:
https://github.com/geoserver/geoserver/pull/2140
Opinions on this ?

Travis CI reports many app-schema unit test failures:
https://s3.amazonaws.com/archive.travis-ci.org/jobs/207942836/log.txt

Kind regards,

--

GeoServer Professional Services from the experts!
Visit http://goo.gl/it488V for more information.

Nuno Miguel Carvalho Oliveira
@nmcoliveira
Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono
da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate
nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e
-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo
anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.
  The information in this message and/or attachments, is intended solely for the attention and use of
the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree
June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying,
distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender does not give any warranty or accept liability as the content,
accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which
arise as a result of e-mail transmission, viruses, etc.

Hi,
just a quick addition, another problem of the current approach is that it breaks security, if a user
cannot see a certain attribute and the schema gets built with that user making the request,
no other user will ever see that attribute either in GML

Cheers
Andrea

···

On Sun, Mar 5, 2017 at 6:21 PM, Nuno Oliveira <nuno.oliveira@anonymised.com> wrote:

Hi,

Please check this issue description: https://osgeo-org.atlassian.net/browse/GEOS-8017

So basically the idea is to stop using a global cached schema (for all the feature types) in WFS 1.1.0 but instead build a specific schema for each request using only the necessary feature types similar to what is done in WFS 2.0.0.

Another advantage is that we will also avoid the cost of building a global schema for all the feature types which can take some time when we have a lot of feature types.

I made some performance tests to measure the impact of this solution. I used two GeoServer instances:

  • NORMAL GEOSERVER - 8076 - normal GeoServer that ues a global cached schema

  • CHANGED GEOSERVER - 8075 - changed GeoServer that uses the solution described above

I used three (vector) layers for doing this tests:

  • sf:roads - shipped with GeoServer release
  • topp:roads - shipped with GeoServer release
  • it.geosolutions:cache_test - please find it attached, shapefile with pints that have 30 attributes (integer, string and date)

Each layer features (MAX 3 features per request) were requested using WFS 1.1.0 and WFS 2.0.0 on each GeoServer instance (WFS 2.0.0 results can be used as control values). I run this tests using JMeter (please find the JMeter configuration attached), the results are as follows:

My conclusion is that the proposed solution has an insignificant performance penalty, the throughput is the same for all the cases. I also monitored the two GeoServer instances JVMs using VisualVM:

The CPU usage is a little bit higher in the changed instance and no memory leaks were detected.

Since the above solutions touches how schemas are generated I (try to) run the CITE tests for both instances. Long story short, I was no able to successfully run the tests suites that requires test data to be instantiated (lot’s of errors during the tests). The results are equal on both servers unfortunately with the same tests failed on both:

NORMAL SERVER:

CHANGED SERVER:

A pull request for this was made: https://github.com/geoserver/geoserver/pull/2140

Opinions on this ?

Kind regards,

Nuno Oliveira

-- 
==
GeoServer Professional Services from the experts! 
Visit [http://goo.gl/it488V](http://goo.gl/it488V) for more information.
==
Nuno Miguel Carvalho Oliveira
@nmcoliveira
Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy

phone: +39 0584 962313
fax:   +39 0584 1660272
mob:   +39  333 8128928

[http://www.geo-solutions.it](http://www.geo-solutions.it)
[http://twitter.com/geosolutions_it](http://twitter.com/geosolutions_it)

-------------------------------------------------------

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono
da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate
nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e
-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo
anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.
 
The information in this message and/or attachments, is intended solely for the attention and use of
the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree
June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying,
distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender does not give any warranty or accept liability as the content,
accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which
arise as a result of e-mail transmission, viruses, etc.

Check out the vibrant tech community on one of the world’s most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot


Geoserver-devel mailing list
Geoserver-devel@anonymised.com.366…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


I tried an local merge of the amended pull request and a full build and app-schema online tests both pass.

Kind regards,
Ben.

On 07/03/17 03:08, Nuno Oliveira wrote:

Hi Ben,

Thanks for the warning, pull request amended.

Last minute change and forget to check app-schema tests which proved
that my last minute change was actually a bad idea for when schemas are
provided by the feature type :frowning:

Kind regards,

Nuno Oliveira

On 03/05/2017 08:10 PM, Ben Caradoc-Davies wrote:

On 06/03/17 06:21, Nuno Oliveira wrote:

A pull request for this was made:
https://github.com/geoserver/geoserver/pull/2140
Opinions on this ?

Travis CI reports many app-schema unit test failures:
https://s3.amazonaws.com/archive.travis-ci.org/jobs/207942836/log.txt

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand