[Geoserver-devel] Speaking of slow startup time and speeding it up...

GeoServer GeoServer Developer
1

Hi,
while I had my head on the topic I’ve put together the following pull request (careful, it’s for discussion, not to be merged):

https://github.com/geoserver/geoserver/pull/1999

The idea is that store checks are there only to protect the capabilities document generation from exceptions, but if
we have the caps generation style configured to be tolerant to invalid layers and skip over them, then the check
is not required.

Implementation wise, catalog is loaded before the service configuration and global.xml in particular, but I’ve
added code to have a sneak peak at it before loading the catalog.

Also, GeoServerInfoImpl has been modified to default on tolerant generation if no flag is set.

What do you think, worth pursuing?

Cheers
Andrea

···

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

2

Startup time is always worth pursuing; more so now with containers and provisioning scripts… not checking all the datastore connections during startup sounds like a great time savings. To think this through when would layers be checked GetCapabilities should be able to generate without checking datastore connections; I would expect only layer operations (DescribeLayer, GetMap, GetFeatureInfo) to force a datastore connection …

I would of expected global to be loaded before catalog and services; but perhaps (global is just for settings that are common to services). During the code sprint we split global into two sections almost along these lines “ogc services” and “internal settings”.

···

On 4 December 2016 at 07:47, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
while I had my head on the topic I’ve put together the following pull request (careful, it’s for discussion, not to be merged):

https://github.com/geoserver/geoserver/pull/1999

The idea is that store checks are there only to protect the capabilities document generation from exceptions, but if
we have the caps generation style configured to be tolerant to invalid layers and skip over them, then the check
is not required.

Implementation wise, catalog is loaded before the service configuration and global.xml in particular, but I’ve
added code to have a sneak peak at it before loading the catalog.

Also, GeoServerInfoImpl has been modified to default on tolerant generation if no flag is set.

What do you think, worth pursuing?

Cheers
Andrea

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

Check out the vibrant tech community on one of the world’s most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot

Geoserver-devel mailing list
Geoserver-devel@anonymised.com.366…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–
Jody Garnett

3

On Tue, Dec 6, 2016 at 11:13 PM, Jody Garnett <jody.garnett@anonymised.com>
wrote:

Startup time is always worth pursuing; more so now with containers and
provisioning scripts... not checking all the datastore connections during
startup sounds like a great time savings. To think this through when would
layers be checked GetCapabilities should be able to generate without
checking datastore connections; I would expect only layer operations
(DescribeLayer, GetMap, GetFeatureInfo) to force a datastore connection ...

The expectation is not matching reality at the moment I'm afraid. At least
the WMS capabilites document generator needs to know if a certain layer is
geometryless or not, that at the moment can only be
done by grabbing the schema, which requires opening a connection. If you
have interest in this, you could explore an option to mark geometryless
layers on creation/modification, however, that would still fail if a layer
with geometry becomes geometryless later (as a result of a database change
for example).

I would of expected global to be loaded before catalog and services; but
perhaps (global is just for settings that are common to services). During
the code sprint we split global into two sections almost along these lines
"ogc services" and "internal settings".

Nope, services have always been loaded after the catalog, you should not
form an idea just by looking at the UI when the code is there ready for the
reading :wink:

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

4

Hi Andrea,

···

On 8 December 2016 at 10:00, Andrea Aime <andrea.aime@anonymised.com> wrote:

Fundamentally most things don’t change most of the time, so wouldn’t saving it at creation/modification be a better approach in general? For WFS the same thing effectively happens where Geoserver saves the DB fields/etc against the FeatureType definition…

Cheers,

Rob :slight_smile:

At least the WMS capabilites document generator needs to know if a certain layer is geometryless or not, that at the moment can only be
done by grabbing the schema, which requires opening a connection. If you have interest in this, you could explore an option to mark geometryless layers on creation/modification, however, that would still fail if a layer
with geometry becomes geometryless later (as a result of a database change for example).

On 8 December 2016 at 10:00, Andrea Aime <andrea.aime@anonymised.com> wrote:

Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today.http://sdm.link/xeonphi

Geoserver-devel mailing list
Geoserver-devel@anonymised.com.366…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

On Tue, Dec 6, 2016 at 11:13 PM, Jody Garnett <jody.garnett@anonymised.com> wrote:

Startup time is always worth pursuing; more so now with containers and provisioning scripts… not checking all the datastore connections during startup sounds like a great time savings. To think this through when would layers be checked GetCapabilities should be able to generate without checking datastore connections; I would expect only layer operations (DescribeLayer, GetMap, GetFeatureInfo) to force a datastore connection …

The expectation is not matching reality at the moment I’m afraid. At least the WMS capabilites document generator needs to know if a certain layer is geometryless or not, that at the moment can only be
done by grabbing the schema, which requires opening a connection. If you have interest in this, you could explore an option to mark geometryless layers on creation/modification, however, that would still fail if a layer
with geometry becomes geometryless later (as a result of a database change for example).

I would of expected global to be loaded before catalog and services; but perhaps (global is just for settings that are common to services). During the code sprint we split global into two sections almost along these lines “ogc services” and “internal settings”.

Nope, services have always been loaded after the catalog, you should not form an idea just by looking at the UI when the code is there ready for the reading :wink:

Cheers

Andrea

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

Chief Technology Officer Koordinates

+44 759 987 3480 / koordinates.com / @koordinates

5

On Fri, Dec 9, 2016 at 9:26 AM, Robert Coup <robert.coup@anonymised.com>
wrote:

Hi Andrea,

On 8 December 2016 at 10:00, Andrea Aime <andrea.aime@anonymised.com>
wrote:

At least the WMS capabilites document generator needs to know if a
certain layer is geometryless or not, that at the moment can only be
done by grabbing the schema, which requires opening a connection. If you
have interest in this, you could explore an option to mark geometryless
layers on creation/modification, however, that would still fail if a layer
with geometry becomes geometryless later (as a result of a database
change for example).

Fundamentally most things don't change most of the time, so wouldn't
saving it at creation/modification be a better approach in general? For WFS
the same thing effectively happens where Geoserver saves the DB fields/etc
against the FeatureType definition...

Except that it does not, they are always retrieved "sort of fresh", what
you see in the REST calls is coming from an in memory cache of feature
types held in the ResourcePool, the same that
the WMS generator uses to determine if something is geometryless or not.
It's just that a caps generation tends to thrash the cache (same does an
unqualified
DescribeFeatureType), to avoid it one can set a higher value for the
"feature type cache":
http://docs.geoserver.org/stable/en/user/configuration/globalsettings.html#feature-type-cache-size

Btw, legacy featuretype.xml still carry the attribute list (like the ones
we have in our release directory), but the ones created
in the last few years should not instead. I believe nowadays if you
actually manually set that list, it will be treated as an way to
filter the actual underlying attributes (e.g., if an attribute is not in
that list, it's not visible)... and I believe the only way to
actually set that list is to use the REST API.

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

6

Hi all,

···

On 9 December 2016 at 10:39, Andrea Aime <andrea.aime@anonymised.com268…> wrote:

TIL - in fact we do that always via REST, always set an attribute list (and use it as a filter on attributes), but I assumed the UI behaved similarly. Guess not!

Rob :slight_smile:

Btw, legacy featuretype.xml still carry the attribute list (like the ones we have in our release directory), but the ones created in the last few years should not instead. I believe nowadays if you actually manually set that list, it will be treated as an way to filter the actual underlying attributes (e.g., if an attribute is not in that list, it’s not visible)… and I believe the only way to actually set that list is to use the REST API.

7

On Tue, Dec 13, 2016 at 11:28 AM, Robert Coup <robert.coup@anonymised.com>
wrote:

TIL - in fact we do that always via REST, always set an attribute list
(and use it as a filter on attributes), but I assumed the UI behaved
similarly. Guess not!

Eh, I'm actually pretty confused myself... looking at the test data
directory that I've created to test speed, with the importer,
all attributes are indeed saved in the featuretype.xml... this also
contributes to a fast speedup, because we have a validator
that checks if there is a geometry attribute, and this would be slow if the
list was not there to start with.

However, that also implies that changing the database structure would not
be picked up by GeoServer, not reloading it, not
even restarting it... and apparently not even hitting "reload feature type"
from the UI...
There is something going on here that I don't fully understand....

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------