[Geoserver-devel] Slow build, security setup an important portion of our build times

GeoServer GeoServer Developer
1

Hi,
I’ve been spending some time looking as to why our tests are so slow, so I’ve profiled
a WFS surefire process while it was running our tests.

Here is a breakdown from the main method:

root) org.apache.maven.surefire.booter.ForkedBooter.main(String) 94094 0

…
setup) org.geoserver.test.GeoServerBaseTestSupport.doSetup() 49714 0

test-methods) sun.reflect.NativeMethodAccessorImpl.invoke0(Method, Object, Object) 35130 0

The test-methods root contains all the single test methods runs, e.g.:

  • org.geoserver.wfs.ExternalEntitiesTest.testWfs1_1() 932 0

  • org.geoserver.wfs.AliasTest.testAliasFifteen() 856 0

  • org.geoserver.wfs.GetFeaturePagingTest.testStartIndexMultipleTypesPOST() 755 0

  • org.geoserver.wfs.GetFeaturePagingTest.testStartIndexMultipleTypes() 584 0

and so on

So… we are spending 53% of our build time running the various setups… not nice.
And here is how the doSetup breaks down:

So, we are spending 32% of our overall build time running security subsystem data directory migrations…
that really needs to be fixed, I believe, it’s pretty much the same time we spend actually running
test methods…

I guess the best approach would be to make sure that we generate the security subsystem config
already at the latest version required.
But, there is something else going on here, those migration result in a insane amount of calls to
the GeoServerSecuritManager.persister(), which in turns creates a XStream persister from
scratch. Have a look, we call that method 300.000 times during the WFS test build alone:

I believe there is something fishy going on there, like, I see the HelperBase.loadConfig class
called over and over loading the same files, I’m afraid there might be some sort of event escalation
(like, changing one of the security config files, which results in the whole security being reloaded,
then change another, and so on).

The code seems in need of a refactor, where a single persister is reused for all the migration operations.

Christian, do you have some time to look into this issue? By comparisong, loading the catalog
from the many little config files takes only 3 seconds overall, that is, 1/10th of the time we spend
doing these migrations.

Cheers
Andrea
​

···

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

2

Hi Andrea

Would be a good idea to have a migrated security configuration on master, 2.7.x and 2.6.x. I think we can use the identical configuration on each branch.

I have no idea about the HelperBase class, I think it was introduced by Mauro or Justin.

Cheers
Christian

(attachments)

Selezione_017.png
Selezione_016.png

···

On Sun, Apr 19, 2015 at 4:29 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
I’ve been spending some time looking as to why our tests are so slow, so I’ve profiled
a WFS surefire process while it was running our tests.

Here is a breakdown from the main method:

root) org.apache.maven.surefire.booter.ForkedBooter.main(String) 94094 0

…
setup) org.geoserver.test.GeoServerBaseTestSupport.doSetup() 49714 0

test-methods) sun.reflect.NativeMethodAccessorImpl.invoke0(Method, Object, Object) 35130 0

The test-methods root contains all the single test methods runs, e.g.:

  • org.geoserver.wfs.ExternalEntitiesTest.testWfs1_1() 932 0

  • org.geoserver.wfs.AliasTest.testAliasFifteen() 856 0

  • org.geoserver.wfs.GetFeaturePagingTest.testStartIndexMultipleTypesPOST() 755 0

  • org.geoserver.wfs.GetFeaturePagingTest.testStartIndexMultipleTypes() 584 0

and so on

So… we are spending 53% of our build time running the various setups… not nice.
And here is how the doSetup breaks down:

So, we are spending 32% of our overall build time running security subsystem data directory migrations…
that really needs to be fixed, I believe, it’s pretty much the same time we spend actually running
test methods…

I guess the best approach would be to make sure that we generate the security subsystem config
already at the latest version required.
But, there is something else going on here, those migration result in a insane amount of calls to
the GeoServerSecuritManager.persister(), which in turns creates a XStream persister from
scratch. Have a look, we call that method 300.000 times during the WFS test build alone:

I believe there is something fishy going on there, like, I see the HelperBase.loadConfig class
called over and over loading the same files, I’m afraid there might be some sort of event escalation
(like, changing one of the security config files, which results in the whole security being reloaded,
then change another, and so on).

The code seems in need of a refactor, where a single persister is reused for all the migration operations.

Christian, do you have some time to look into this issue? By comparisong, loading the catalog
from the many little config files takes only 3 seconds overall, that is, 1/10th of the time we spend
doing these migrations.

Cheers
Andrea
​

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

–

DI Christian Mueller MSc (GIS), MSc (IT-Security)
OSS Open Source Solutions GmbH

3

Hi Andrea

What profiling tool do you use ?. Would make sense if I use it too.

Cheers
Chrilstian

(attachments)

Selezione_017.png
Selezione_016.png

···

On Mon, Apr 20, 2015 at 3:06 PM, Christian Mueller <christian.mueller@anonymised.com> wrote:

Hi Andrea

Would be a good idea to have a migrated security configuration on master, 2.7.x and 2.6.x. I think we can use the identical configuration on each branch.

I have no idea about the HelperBase class, I think it was introduced by Mauro or Justin.

Cheers
Christian

–

On Sun, Apr 19, 2015 at 4:29 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
I’ve been spending some time looking as to why our tests are so slow, so I’ve profiled
a WFS surefire process while it was running our tests.

Here is a breakdown from the main method:

root) org.apache.maven.surefire.booter.ForkedBooter.main(String) 94094 0

…
setup) org.geoserver.test.GeoServerBaseTestSupport.doSetup() 49714 0

test-methods) sun.reflect.NativeMethodAccessorImpl.invoke0(Method, Object, Object) 35130 0

The test-methods root contains all the single test methods runs, e.g.:

  • org.geoserver.wfs.ExternalEntitiesTest.testWfs1_1() 932 0

  • org.geoserver.wfs.AliasTest.testAliasFifteen() 856 0

  • org.geoserver.wfs.GetFeaturePagingTest.testStartIndexMultipleTypesPOST() 755 0

  • org.geoserver.wfs.GetFeaturePagingTest.testStartIndexMultipleTypes() 584 0

and so on

So… we are spending 53% of our build time running the various setups… not nice.
And here is how the doSetup breaks down:

So, we are spending 32% of our overall build time running security subsystem data directory migrations…
that really needs to be fixed, I believe, it’s pretty much the same time we spend actually running
test methods…

I guess the best approach would be to make sure that we generate the security subsystem config
already at the latest version required.
But, there is something else going on here, those migration result in a insane amount of calls to
the GeoServerSecuritManager.persister(), which in turns creates a XStream persister from
scratch. Have a look, we call that method 300.000 times during the WFS test build alone:

I believe there is something fishy going on there, like, I see the HelperBase.loadConfig class
called over and over loading the same files, I’m afraid there might be some sort of event escalation
(like, changing one of the security config files, which results in the whole security being reloaded,
then change another, and so on).

The code seems in need of a refactor, where a single persister is reused for all the migration operations.

Christian, do you have some time to look into this issue? By comparisong, loading the catalog
from the many little config files takes only 3 seconds overall, that is, 1/10th of the time we spend
doing these migrations.

Cheers
Andrea
​

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

–

DI Christian Mueller MSc (GIS), MSc (IT-Security)
OSS Open Source Solutions GmbH

DI Christian Mueller MSc (GIS), MSc (IT-Security)
OSS Open Source Solutions GmbH

4

On Tue, Apr 21, 2015 at 9:37 AM, Christian Mueller <
christian.mueller@anonymised.com> wrote:

Hi Andrea

What profiling tool do you use ?. Would make sense if I use it too.

I have a yourkit open source license, that's what I used here. Not sure who
asked last for the
yourkit licenses though.

I've also played a bit with java mission control, not too bad:
http://hirt.se/blog/?p=364

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

5

Do we have a reference to yourkit on the geoserver home page ?. I think this is a prerequisite for getting an OS license.

Cheers
Christian

···

On Tue, Apr 21, 2015 at 9:42 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

–

On Tue, Apr 21, 2015 at 9:37 AM, Christian Mueller <christian.mueller@anonymised.com> wrote:

Hi Andrea

What profiling tool do you use ?. Would make sense if I use it too.

I have a yourkit open source license, that’s what I used here. Not sure who asked last for the
yourkit licenses though.

I’ve also played a bit with java mission control, not too bad:
http://hirt.se/blog/?p=364

Cheers

Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

DI Christian Mueller MSc (GIS), MSc (IT-Security)
OSS Open Source Solutions GmbH

6

On Tue, Apr 21, 2015 at 12:10 PM, Christian Mueller <
christian.mueller@anonymised.com> wrote:

Do we have a reference to yourkit on the geoserver home page ?. I think
this is a prerequisite for getting an OS license.

We used to have one... it got removed by Jody some time ago:
https://github.com/geoserver/geoserver.github.io/commit/eeed022df9b5124edaf71f7b1336bcf55616d09a

I think that maybe we got the licences though Codehaus that round? Not sure.

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

7

Hi Andrea

I managed to get a yourkit Open Source license key. The key is valid for one installation and I plan to use this key at my home office. Hope to get some time for investigation during the weekend. Btw, can you tell me the test case(s) according to your screen shots.

Cheers
Chrilstian

···

On Tue, Apr 21, 2015 at 12:41 PM, Andrea Aime <andrea.aime@anonymised.com68…> wrote:

–

On Tue, Apr 21, 2015 at 12:10 PM, Christian Mueller <christian.mueller@anonymised.com> wrote:

Do we have a reference to yourkit on the geoserver home page ?. I think this is a prerequisite for getting an OS license.

We used to have one… it got removed by Jody some time ago:
https://github.com/geoserver/geoserver.github.io/commit/eeed022df9b5124edaf71f7b1336bcf55616d09a

I think that maybe we got the licences though Codehaus that round? Not sure.

Cheers

Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

DI Christian Mueller MSc (GIS), MSc (IT-Security)
OSS Open Source Solutions GmbH

8

On Wed, Apr 22, 2015 at 1:25 PM, Christian Mueller <
christian.mueller@anonymised.com> wrote:

Hi Andrea

I managed to get a yourkit Open Source license key. The key is valid for
one installation and I plan to use this key at my home office. Hope to get
some time for investigation during the weekend. Btw, can you tell me the
test case(s) according to your screen shots.

That's not a single test case, I did attach yourkit to the surefire test
JVM while gs-wfs was running.
Anyways, for reference the method that's called 3 million times, I did add
a static counter into it
and have it to a system out every time it was called, running one of the
GetFeatureTests I checked
it got called over 300 times (all during the test setup).

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

9

Hi Andrea

I think the security configuration test data is ok, but the migrate method is called before the the test setup. The security dir is empty and the migration starts. The method

org.geoserver.security.impl.GeoServerJ2eeRoleServiceTest.onSetUp(SystemTestData)

is called after the migration and in my case the following method is never called:

org.geoserver.data.test.SystemTestData.setUpSecurity()

Look at the stack trace

Thread [main] (Suspended (breakpoint at line 2119 in GeoServerSecurityManager))
GeoServerSecurityManager.migrateFrom21() line: 2119
GeoServerSecurityManager.onApplicationEvent(ApplicationEvent) line: 329
SimpleApplicationEventMulticaster.multicastEvent(ApplicationEvent) line: 97
GeoServerTestApplicationContext(AbstractApplicationContext).publishEvent(ApplicationEvent) line: 327
GeoServerJ2eeRoleServiceTest(GeoServerSystemTestSupport).setUp(SystemTestData) line: 208
GeoServerJ2eeRoleServiceTest(GeoServerSystemTestSupport).setUp(TestData) line: 1
GeoServerJ2eeRoleServiceTest(GeoServerBaseTestSupport).doSetup() line: 151
NativeMethodAccessorImpl.invoke0(Method, Object, Object) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object) line: 57
DelegatingMethodAccessorImpl.invoke(Object, Object) line: 43
Method.invoke(Object, Object…) line: 606
FrameworkMethod$1.runReflectiveCall() line: 47
FrameworkMethod$1(ReflectiveCallable).run() line: 12
FrameworkMethod.invokeExplosively(Object, Object…) line: 44
RunBefores.evaluate() line: 24
RunAfters.evaluate() line: 27
RunRules.evaluate() line: 20
BlockJUnit4ClassRunner(ParentRunner).runLeaf(Statement, Description, RunNotifier) line: 271
BlockJUnit4ClassRunner.runChild(FrameworkMethod, RunNotifier) line: 70
BlockJUnit4ClassRunner.runChild(Object, RunNotifier) line: 50
ParentRunner$3.run() line: 238
ParentRunner$1.schedule(Runnable) line: 63
BlockJUnit4ClassRunner(ParentRunner).runChildren(RunNotifier) line: 236
ParentRunner.access$000(ParentRunner, RunNotifier) line: 53
ParentRunner$2.evaluate() line: 229
RunBefores.evaluate() line: 26
RunAfters.evaluate() line: 27
BlockJUnit4ClassRunner(ParentRunner).run(RunNotifier) line: 309
JUnit4TestClassReference(JUnit4TestReference).run(TestExecution) line: 50
TestExecution.run(ITestReference) line: 38
RemoteTestRunner.runTests(String, String, TestExecution) line: 459
RemoteTestRunner.runTests(TestExecution) line: 675
RemoteTestRunner.run() line: 382
RemoteTestRunner.main(String) line: 192

Any idea ?. I do not want to experiment with the test setup, its complicated enough.

Cheers
Christian

···

On Wed, Apr 22, 2015 at 1:59 PM, Andrea Aime <andrea.aime@anonymised.com1268…> wrote:

–

On Wed, Apr 22, 2015 at 1:25 PM, Christian Mueller <christian.mueller@anonymised.com> wrote:

Hi Andrea

I managed to get a yourkit Open Source license key. The key is valid for one installation and I plan to use this key at my home office. Hope to get some time for investigation during the weekend. Btw, can you tell me the test case(s) according to your screen shots.

That’s not a single test case, I did attach yourkit to the surefire test JVM while gs-wfs was running.
Anyways, for reference the method that’s called 3 million times, I did add a static counter into it
and have it to a system out every time it was called, running one of the GetFeatureTests I checked
it got called over 300 times (all during the test setup).

Cheers

Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

DI Christian Mueller MSc (GIS), MSc (IT-Security)
OSS Open Source Solutions GmbH

10

On Fri, Apr 24, 2015 at 4:24 PM, Christian Mueller <
christian.mueller@anonymised.com> wrote:

Hi Andrea

I think the security configuration test data is ok, but the migrate method
is called before the the test setup. The security dir is empty and the
migration starts. The method

But... why do we need to migrate at all if the security dir is empty? :slight_smile:

Regardless, I'll have a look, we should be setting the data dir before
those events are sent around

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

11

On Fri, Apr 24, 2015 at 4:35 PM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

On Fri, Apr 24, 2015 at 4:24 PM, Christian Mueller <
christian.mueller@anonymised.com> wrote:

Hi Andrea

I think the security configuration test data is ok, but the migrate
method is called before the the test setup. The security dir is empty and
the migration starts. The method

Eh, you are not looking at the same tests I suggested you to investigate
(the wfs one), all the
AbstractSecurityServiceTest inherit this one:

    @Override
    protected void setUpTestData(SystemTestData testData) throws Exception {
        //explictily do nothing, we want no layers
    }

Thus, no security directory is generated, and we go though all the
migrations.
I've changed the method to look like:

@Override
    protected void setUpTestData(SystemTestData testData) throws Exception {
        testData.setUpSecurity();
    }

and this way migrateFrom21 and migrateFrom22 exit immediately,
it's migrateFrom23 and migrateFrom24
that are still running (and killing us), just like in other tests.

I'm going to check if I can generate a valid and current security.zip we
can start from to avoid the
expensive setups.

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

12

On Sat, Apr 25, 2015 at 11:54 AM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

I'm going to check if I can generate a valid and current security.zip we
can start from to avoid the
expensive setups.

Hi,
I've tried to take the release data directory of GeoServer, start
GeoServer, and observed the 23 and 24
migrations run and take their time to do so.
Then replaced security.zip contents with the result, and looked at the
tests.
It looks that migrate23 is now skipped, but migrate24 does not have any
skipping logic, and just
runs at every test/startup, even if it's changing nothing.

I believe this one must be fixed?

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

13

On Sat, Apr 25, 2015 at 12:07 PM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

Then replaced security.zip contents with the result, and looked at the
tests.
It looks that migrate23 is now skipped, but migrate24 does not have any
skipping logic, and just
runs at every test/startup, even if it's changing nothing.

Simple idea, can we put a version.properties/version.txt in the security
dir, containing the current
version of the security dir contents (it would be 2.5 now).
The first time we don't find it, go though the migrations, then create it,
from that point on no need
to go though the migrations at all.

For data directories that miss completely a security setup (as it happens
in some tests), wondering,
can't we just use the security.zip of the tests, unpack it, and move on?

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

14

On Sat, Apr 25, 2015 at 9:24 PM, Andrea Aime <andrea.aime@anonymised.com>
wrote:

Simple idea, can we put a version.properties/version.txt in the security
dir, containing the current
version of the security dir contents (it would be 2.5 now).
The first time we don't find it, go though the migrations, then create it,
from that point on no need
to go though the migrations at all.

For data directories that miss completely a security setup (as it happens
in some tests), wondering,
can't we just use the security.zip of the tests, unpack it, and move on?

Hi,
I've played a bit with this idea, here is a branch that applies it and
indeed speeds up the build:
https://github.com/aaime/geoserver/tree/build_security_speedup
In particular, this commit:
https://github.com/aaime/geoserver/commit/ac4a915bfa238f7978efac3faff52eeed1267d2c

Simple partial build times comparison, only up to the wfs module.
Before:

[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] GeoServer .......................................... SUCCESS [
1.630 s]
[INFO] Core Platform Module ............................... SUCCESS [
10.997 s]
[INFO] Open Web Service Module ............................ SUCCESS [
7.582 s]
[INFO] Main Module ........................................ SUCCESS [02:16
min]
[INFO] Web Feature Service Module ......................... SUCCESS [01:39
min]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 04:18 min

After:

[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] GeoServer .......................................... SUCCESS [
1.358 s]
[INFO] Core Platform Module ............................... SUCCESS [
10.543 s]
[INFO] Open Web Service Module ............................ SUCCESS [
7.476 s]
[INFO] Main Module ........................................ SUCCESS [01:50
min]
[INFO] Web Feature Service Module ......................... SUCCESS [01:15
min]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 03:26 min

Overall, a significant speedup.

I've tried a full build too, but if fails in a couple of web-sec-core tests
and I'm having little
luck fixing it them: RoleListPageTest.testRemove
and ConfirmRemovalRolePanelTest.testRemoveRole

RoleListPageTest.testRemove apparently tries to remove all roles from a
list,
and in the normal version, only finds NEW_ROLE to select and delete, but
with
the patched version, the table also contains ROLE_ADMINISTRATOR, whose
removal makes a test fail (wondering, should this role be even removable?).

I haven't investigated the other failure.

Christian, do you have any clue?

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------