[Geoserver-devel] Spring upgrade

GeoServer GeoServer Developer
1

Hi,

I am the maintainer of the MapML module. In testing 2.23-RC1, I noticed a long delay that occurs during processing of pretty simple requests to the module. I have checked out and built the HEAD of the 2.22.x branch, and the delay does not occur there. Since the MapML module is a Spring REST controller, I was wondering if it might have been negatively impacted by a spring upgrade?

Definitely willing to change the code to play nice with the new spring.

Any advice on how to test / evaluate this is welcome.

Thank you

2

Peter:

Thanks for noticing the regression; the spring upgrade is indeed the big internal change this release cycle. Are you in position to do any profiling to determine where the slowdown is occurring? You may be able to use git bisect to determine the exact change that introduced the slowdown.

I would like to schedule the 2.23.0 release later this month.

···

–
Jody Garnett

3

Hi Peter,
can you run a few jstack during the slowdown? Would help us figure out what is slowing it down.

Has anyone else noticed slowdowns in other parts of GeoServer yet?

Cheers

Andrea

···

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts!

Visit http://bit.ly/gs-services-us for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions Group
phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

https://www.geosolutionsgroup.com/

http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail

4

Hi

Sorry I didn’t copy the list in my last email, my mistake.

Anyway, I looked at the Spring changes and nothing really stands out to me, although there is some logging on main that didn’t occur in 2.22.x. It might still have something to do with that, but because I noticed in web.xml that text/.* was being compressed by the org.geoserver.filters.GZIPFilter filter, but .js was not, I tried removing the compression on text.* and sure enough the delay disappears from main / problem is gone for mapml previews.

I don’t think it’s desirable to not compress html and css etc, so I will keep looking, but please let me know if anything occurs to you.

Peter

On Mon, Mar 20, 2023 at 12:24 PM Peter Rushforth <peter.rushforth@anonymised.com> wrote:

A couple of logs with GEOSERVER_DEVELOPER_LOGGING enabled, the first with a (custom) build from the 2.22.x branch. The customization is that I include some geoserver.mapml logging statements. The second is from main today.
They compare pretty closely, except for the big pause. One difference I’ve noticed is the 2.22.x branch’s:

“[filter.GeoServerSecurityContextPersistenceFilter$1] - SecurityContextHolder now cleared, as request processing completed”

log statements don’t have any parallel in today’s 2.24.SNAPSHOT (main) log, so I’m wondering if there is some resource that is being withheld for many / all the resources.

Anyway, will keep looking.

Thanks
Peter

On Mon, Mar 20, 2023 at 11:14 AM Peter Rushforth <peter.rushforth@anonymised.com> wrote:

Hi Andrea, and Jody,

I am running a build off main today, as it has the same behaviour as 2.23-RC1.

I ran a few jstack commands during execution of the tomcat process id, first at idle (jstack-idle.out), then during a mapml preview request (jstack-mapml-out to jstack-mapml.out12) and a couple during just the mapml.css request at the URL bar (jstack-css.out).

I attach these files, hopefully the list will allow them. They don’t tell me much, but if you can spot something, please let me know.

I will see if I can profile the app during the same events, for comparison.

Thanks for your help!
Peter

On Mon, Mar 20, 2023 at 6:48 AM Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi Peter,
can you run a few jstack during the slowdown? Would help us figure out what is slowing it down.

Has anyone else noticed slowdowns in other parts of GeoServer yet?

Cheers

Andrea

On Mon, Mar 20, 2023 at 8:33 AM Jody Garnett <jody.garnett@anonymised.com> wrote:

Peter:

Thanks for noticing the regression; the spring upgrade is indeed the big internal change this release cycle. Are you in position to do any profiling to determine where the slowdown is occurring? You may be able to use git bisect to determine the exact change that introduced the slowdown.

I would like to schedule the 2.23.0 release later this month.

–
Jody Garnett

On Fri, Mar 17, 2023 at 7:38 AM Peter Rushforth <peter.rushforth@anonymised.com> wrote:

Hi,

I am the maintainer of the MapML module. In testing 2.23-RC1, I noticed a long delay that occurs during processing of pretty simple requests to the module. I have checked out and built the HEAD of the 2.22.x branch, and the delay does not occur there. Since the MapML module is a Spring REST controller, I was wondering if it might have been negatively impacted by a spring upgrade?

Definitely willing to change the code to play nice with the new spring.

Any advice on how to test / evaluate this is welcome.

Thank you

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts!

Visit http://bit.ly/gs-services-us for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions Group
phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

https://www.geosolutionsgroup.com/

http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail

5

If that’s the case, then the changes to GSIPResponseOutputStream might be the cause?
But I don’t see how: https://github.com/geoserver/geoserver/pull/6515/files#diff-40e2171fa39da85e5fd855ddfab206e350b990149834a4f16c1e64e0ee72cb24

The upgrade of Spring version required an update to the servet api itself, which caused the two extra methods
to be added. Maybe there is some behavioral change there?

I’ve also grepped the various jstacks attached for any trace of GeoServer activity, but came out empty, e.g:

grep -i “org.geoserver” *

returns nothing… It looks like GeoServer was completely inactive while they were taken.
I’ve also checked out other compressible output formats, like HTML or GeoJSON, and I’m seeing no slowdown, e.g. (demo server on the main branch here):

https://gs-main.geosolutionsgroup.com/geoserver/ogc/features/v1/collections/topp:states/items?f=text%2Fhtml

https://gs-main.geosolutionsgroup.com/geoserver/ogc/features/v1/collections/topp:states/items?f=json

Is there anything else you have going on in the demo server? Changes in web.xml? Other plugins?
Anything special about how mapml writes onto the output?

Cheers
Andrea

···

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts!

Visit http://bit.ly/gs-services-us for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions Group
phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

https://www.geosolutionsgroup.com/

http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail

6

I have isolated the problem to text/.* Content-types, but only as these apply to the /geoserver/mapml/ controller; /geoserver/openlayers/*.css or text/html responses are not affected by the following problem:

some part of the call stack is setting the Content-length header to the size of the ungzipped response. As a result, the client is waiting for the keep-alive timeout value (20 seconds) then closing and rendering the response.

I can’t figure out how to prevent the header from being set, even by commenting out the many places in the GZIPResponseWrapper where it may be set. It gets set outside the call into the GZIPFilter.

Nor can I figure out how to set the header value to the correct figure which would be the size of the compressed response.

I can operationally work around this by not compressing HTML and CSs, but it will bug me, and be trouble for users.

I will continue to try to find a fix.

Cheers
Peter

On Tue, Mar 21, 2023, 5:13 a.m. Andrea Aime <andrea.aime@anonymised.com> wrote:

If that’s the case, then the changes to GSIPResponseOutputStream might be the cause?
But I don’t see how: https://github.com/geoserver/geoserver/pull/6515/files#diff-40e2171fa39da85e5fd855ddfab206e350b990149834a4f16c1e64e0ee72cb24

The upgrade of Spring version required an update to the servet api itself, which caused the two extra methods
to be added. Maybe there is some behavioral change there?

I’ve also grepped the various jstacks attached for any trace of GeoServer activity, but came out empty, e.g:

grep -i “org.geoserver” *

returns nothing… It looks like GeoServer was completely inactive while they were taken.
I’ve also checked out other compressible output formats, like HTML or GeoJSON, and I’m seeing no slowdown, e.g. (demo server on the main branch here):

https://gs-main.geosolutionsgroup.com/geoserver/ogc/features/v1/collections/topp:states/items?f=text%2Fhtml

https://gs-main.geosolutionsgroup.com/geoserver/ogc/features/v1/collections/topp:states/items?f=json

Is there anything else you have going on in the demo server? Changes in web.xml? Other plugins?
Anything special about how mapml writes onto the output?

Cheers
Andrea

On Mon, Mar 20, 2023 at 9:41 PM Peter Rushforth <peter.rushforth@anonymised.com> wrote:

Hi

Sorry I didn’t copy the list in my last email, my mistake.

Anyway, I looked at the Spring changes and nothing really stands out to me, although there is some logging on main that didn’t occur in 2.22.x. It might still have something to do with that, but because I noticed in web.xml that text/.* was being compressed by the org.geoserver.filters.GZIPFilter filter, but .js was not, I tried removing the compression on text.* and sure enough the delay disappears from main / problem is gone for mapml previews.

I don’t think it’s desirable to not compress html and css etc, so I will keep looking, but please let me know if anything occurs to you.

Peter

On Mon, Mar 20, 2023 at 12:24 PM Peter Rushforth <peter.rushforth@…403…> wrote:

A couple of logs with GEOSERVER_DEVELOPER_LOGGING enabled, the first with a (custom) build from the 2.22.x branch. The customization is that I include some geoserver.mapml logging statements. The second is from main today.
They compare pretty closely, except for the big pause. One difference I’ve noticed is the 2.22.x branch’s:

“[filter.GeoServerSecurityContextPersistenceFilter$1] - SecurityContextHolder now cleared, as request processing completed”

log statements don’t have any parallel in today’s 2.24.SNAPSHOT (main) log, so I’m wondering if there is some resource that is being withheld for many / all the resources.

Anyway, will keep looking.

Thanks
Peter

On Mon, Mar 20, 2023 at 11:14 AM Peter Rushforth <peter.rushforth@anonymised.com> wrote:

Hi Andrea, and Jody,

I am running a build off main today, as it has the same behaviour as 2.23-RC1.

I ran a few jstack commands during execution of the tomcat process id, first at idle (jstack-idle.out), then during a mapml preview request (jstack-mapml-out to jstack-mapml.out12) and a couple during just the mapml.css request at the URL bar (jstack-css.out).

I attach these files, hopefully the list will allow them. They don’t tell me much, but if you can spot something, please let me know.

I will see if I can profile the app during the same events, for comparison.

Thanks for your help!
Peter

On Mon, Mar 20, 2023 at 6:48 AM Andrea Aime <andrea.aime@anonymised.com.6887…> wrote:

Hi Peter,
can you run a few jstack during the slowdown? Would help us figure out what is slowing it down.

Has anyone else noticed slowdowns in other parts of GeoServer yet?

Cheers

Andrea

On Mon, Mar 20, 2023 at 8:33 AM Jody Garnett <jody.garnett@anonymised.com3…> wrote:

Peter:

Thanks for noticing the regression; the spring upgrade is indeed the big internal change this release cycle. Are you in position to do any profiling to determine where the slowdown is occurring? You may be able to use git bisect to determine the exact change that introduced the slowdown.

I would like to schedule the 2.23.0 release later this month.

–
Jody Garnett

On Fri, Mar 17, 2023 at 7:38 AM Peter Rushforth <peter.rushforth@anonymised.com…> wrote:

Hi,

I am the maintainer of the MapML module. In testing 2.23-RC1, I noticed a long delay that occurs during processing of pretty simple requests to the module. I have checked out and built the HEAD of the 2.22.x branch, and the delay does not occur there. Since the MapML module is a Spring REST controller, I was wondering if it might have been negatively impacted by a spring upgrade?

Definitely willing to change the code to play nice with the new spring.

Any advice on how to test / evaluate this is welcome.

Thank you

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts!

Visit http://bit.ly/gs-services-us for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions Group
phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

https://www.geosolutionsgroup.com/

http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail

–

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts!

Visit http://bit.ly/gs-services-us for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions Group
phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

https://www.geosolutionsgroup.com/

http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail

7

Hi Peter,
it’s odd, we have other parts of GeoServer that return HTML out of Spring MVC controllers without setting
the content-length, e.g.:

https://gs-main.geosolutionsgroup.com/geoserver/ogc/features/v1/collections?f=text%2Fhtml

It is HTML, gzip compressed, and yet does not have a content-length header. Same goes for the administrative
REST API (I don’t have a link for that one, for obvious reasons).

Checking the controllers set up, the only difference I see is that the MapML ones have the CrossOrigin annotation,
but that seems unrelated…

Cheers
Andrea

···

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts!

Visit http://bit.ly/gs-services-us for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions Group
phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

https://www.geosolutionsgroup.com/

http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail

8

Hi Andrea,

yep, it’s really odd. It seems that no matter what I do to the GZIPFilter and it’s components, the Content-Length header gets set, and committed, outside of the actions of the filter itself. Also, although the web.xml pattern for text/.* should match the text/mapml media type, and I do see it enter the filter, what gets received by the client is not encoded. All of this only happens, as far as I can tell, in URLs within the /geoserver/mapml controller paths. Outside that, as you noticed, the gzipping works / doesn’t work as specified by the filter config in web.xml.

I have to give up the chase at this point, because I’m out of ideas and time. I will make an amendment to the user guide that notes to users that they must remove the gzip filter from web.xml if they want to use the extension. Eventually I’m certain more information will arrive and I’ll be able to fix it.

Thanks for your help!!
Peter

On Thu, Mar 23, 2023 at 5:00 AM Andrea Aime <andrea.aime@anonymised.com887…> wrote:

Hi Peter,
it’s odd, we have other parts of GeoServer that return HTML out of Spring MVC controllers without setting
the content-length, e.g.:

https://gs-main.geosolutionsgroup.com/geoserver/ogc/features/v1/collections?f=text%2Fhtml

It is HTML, gzip compressed, and yet does not have a content-length header. Same goes for the administrative
REST API (I don’t have a link for that one, for obvious reasons).

Checking the controllers set up, the only difference I see is that the MapML ones have the CrossOrigin annotation,
but that seems unrelated…

Cheers
Andrea

On Thu, Mar 23, 2023 at 4:47 AM Peter Rushforth <peter.rushforth@anonymised.com> wrote:

I have isolated the problem to text/.* Content-types, but only as these apply to the /geoserver/mapml/ controller; /geoserver/openlayers/*.css or text/html responses are not affected by the following problem:

some part of the call stack is setting the Content-length header to the size of the ungzipped response. As a result, the client is waiting for the keep-alive timeout value (20 seconds) then closing and rendering the response.

I can’t figure out how to prevent the header from being set, even by commenting out the many places in the GZIPResponseWrapper where it may be set. It gets set outside the call into the GZIPFilter.

Nor can I figure out how to set the header value to the correct figure which would be the size of the compressed response.

I can operationally work around this by not compressing HTML and CSs, but it will bug me, and be trouble for users.

I will continue to try to find a fix.

Cheers
Peter

On Tue, Mar 21, 2023, 5:13 a.m. Andrea Aime <andrea.aime@anonymised.com> wrote:

If that’s the case, then the changes to GSIPResponseOutputStream might be the cause?
But I don’t see how: https://github.com/geoserver/geoserver/pull/6515/files#diff-40e2171fa39da85e5fd855ddfab206e350b990149834a4f16c1e64e0ee72cb24

The upgrade of Spring version required an update to the servet api itself, which caused the two extra methods
to be added. Maybe there is some behavioral change there?

I’ve also grepped the various jstacks attached for any trace of GeoServer activity, but came out empty, e.g:

grep -i “org.geoserver” *

returns nothing… It looks like GeoServer was completely inactive while they were taken.
I’ve also checked out other compressible output formats, like HTML or GeoJSON, and I’m seeing no slowdown, e.g. (demo server on the main branch here):

https://gs-main.geosolutionsgroup.com/geoserver/ogc/features/v1/collections/topp:states/items?f=text%2Fhtml

https://gs-main.geosolutionsgroup.com/geoserver/ogc/features/v1/collections/topp:states/items?f=json

Is there anything else you have going on in the demo server? Changes in web.xml? Other plugins?
Anything special about how mapml writes onto the output?

Cheers
Andrea

On Mon, Mar 20, 2023 at 9:41 PM Peter Rushforth <peter.rushforth@anonymised.com> wrote:

Hi

Sorry I didn’t copy the list in my last email, my mistake.

Anyway, I looked at the Spring changes and nothing really stands out to me, although there is some logging on main that didn’t occur in 2.22.x. It might still have something to do with that, but because I noticed in web.xml that text/.* was being compressed by the org.geoserver.filters.GZIPFilter filter, but .js was not, I tried removing the compression on text.* and sure enough the delay disappears from main / problem is gone for mapml previews.

I don’t think it’s desirable to not compress html and css etc, so I will keep looking, but please let me know if anything occurs to you.

Peter

On Mon, Mar 20, 2023 at 12:24 PM Peter Rushforth <peter.rushforth@anonymised.com3…> wrote:

A couple of logs with GEOSERVER_DEVELOPER_LOGGING enabled, the first with a (custom) build from the 2.22.x branch. The customization is that I include some geoserver.mapml logging statements. The second is from main today.
They compare pretty closely, except for the big pause. One difference I’ve noticed is the 2.22.x branch’s:

“[filter.GeoServerSecurityContextPersistenceFilter$1] - SecurityContextHolder now cleared, as request processing completed”

log statements don’t have any parallel in today’s 2.24.SNAPSHOT (main) log, so I’m wondering if there is some resource that is being withheld for many / all the resources.

Anyway, will keep looking.

Thanks
Peter

On Mon, Mar 20, 2023 at 11:14 AM Peter Rushforth <peter.rushforth@anonymised.com> wrote:

Hi Andrea, and Jody,

I am running a build off main today, as it has the same behaviour as 2.23-RC1.

I ran a few jstack commands during execution of the tomcat process id, first at idle (jstack-idle.out), then during a mapml preview request (jstack-mapml-out to jstack-mapml.out12) and a couple during just the mapml.css request at the URL bar (jstack-css.out).

I attach these files, hopefully the list will allow them. They don’t tell me much, but if you can spot something, please let me know.

I will see if I can profile the app during the same events, for comparison.

Thanks for your help!
Peter

On Mon, Mar 20, 2023 at 6:48 AM Andrea Aime <andrea.aime@anonymised.com.> wrote:

Hi Peter,
can you run a few jstack during the slowdown? Would help us figure out what is slowing it down.

Has anyone else noticed slowdowns in other parts of GeoServer yet?

Cheers

Andrea

On Mon, Mar 20, 2023 at 8:33 AM Jody Garnett <jody.garnett@anonymised.com> wrote:

Peter:

Thanks for noticing the regression; the spring upgrade is indeed the big internal change this release cycle. Are you in position to do any profiling to determine where the slowdown is occurring? You may be able to use git bisect to determine the exact change that introduced the slowdown.

I would like to schedule the 2.23.0 release later this month.

–
Jody Garnett

On Fri, Mar 17, 2023 at 7:38 AM Peter Rushforth <peter.rushforth@anonymised.com> wrote:

Hi,

I am the maintainer of the MapML module. In testing 2.23-RC1, I noticed a long delay that occurs during processing of pretty simple requests to the module. I have checked out and built the HEAD of the 2.22.x branch, and the delay does not occur there. Since the MapML module is a Spring REST controller, I was wondering if it might have been negatively impacted by a spring upgrade?

Definitely willing to change the code to play nice with the new spring.

Any advice on how to test / evaluate this is welcome.

Thank you

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts!

Visit http://bit.ly/gs-services-us for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions Group
phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

https://www.geosolutionsgroup.com/

http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail

–

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts!

Visit http://bit.ly/gs-services-us for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions Group
phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

https://www.geosolutionsgroup.com/

http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail

–

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts!

Visit http://bit.ly/gs-services-us for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions Group
phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

https://www.geosolutionsgroup.com/

http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail

9

Hi Peter,
had a quick look using the built-in Jetty container… I cannot reproduce the issue, the viewer is always fast.
Debugging, I’ve set breakspoints in the Jetty Response objects dealing with headers, I’ve observed the following:

  • GZIPResponseWrapper/AlternativesResponseStream is holding onto an eventual content-length attribute, by overriding all methods that allowing setting one (addHeader, setHeader, addIntHeader, setIntHeader).
  • It eventually calls the response.setContentLength only if the mime type is found to be non compressible, otherwise, the header won’t be set.
    I could think that maybe a newer version of the Servlet API has other ways to set the header… I cannot find any straight
    method there, but this one caught my eye, and it’s available only in Servlet 4:

https://javadoc.io/static/javax.servlet/javax.servlet-api/4.0.1/javax/servlet/http/HttpServletResponse.html#setTrailerFields-java.util.function.Supplier-

That allows setting headers at the end of response, Content-lenght seems like it would be a good candidate for it.
Now, GeoServer uses Servlet 3.1 where that method is not available, but if you were to deploy GeoServer in Tomcat 10 with the automatic on the fly
servlet API transformation (not recommended!) then I can see how that would escape the current management.
Either that, or I’ve missed places where the content-length can be set, in general, if there is a method that escapes
the wrapping, that would cause the header to be set while compression is activated.

Just thinking out loud, mind!!

Cheers
Andrea

···

Regards,

Andrea Aime

==
GeoServer Professional Services from the experts!

Visit http://bit.ly/gs-services-us for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions Group
phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

https://www.geosolutionsgroup.com/

http://twitter.com/geosolutions_it

Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail