Hi,
I managed to get a “light” paralellization during the catalog load by rolling a background
thread pool that would take care of loading the xml files into memory, without having
to parallelize the code that runs in the spring startup.
Basically, the existing code does a number of “scan this directory for xml files and load them
one by one”, that’s the part I parallelized by creating a asynch iterator doing the loading,
and giving the main thread a byte for each file, to decode and put int the catalog.
The benefit of doing so is visible because IO is the bottleneck.
So, let’s see some numbers, but before, a reminder of the data dirs involved:
- “Many states”: 1 workspace, 1 store, 10k layers, 10k cached layers
- “Large”: 1001 workspaces, 11000 stores (a mix of shapefiles, postgis, directory of shapefile, single tiff, arcgrid, mosaics), 42000 layers and 42000 associated tile layers
Here is a comparison of the last loading times I have vs the ones with parallel IO loading.
- Many states, cold startup. Before, 68s. After: 30s
- Many states, warm startup. Before: 29s After: 21s
- Large, cold startup. Before: 230s After: 107s
- Large, warm startup: Before: 45s After: 45s (weird?)
As expected the benefit shows up mostly on col startups, where actual IO against the disk happens (btw, the
IO is likely happening against the SSD cache of my hybrid HD, a pure spinning disk drive will likely have worse timings).
Pull request available here, please let me know if anybody wants to review or if you have reservations
against a merge (plan B, land it after the freeze and wait September to have it in a release):
https://github.com/geoserver/geoserver/pull/2116
Cheers
Andrea
PS: the IO rate I see is still quite a bit below the potential of my local drive, so a true
parallelization of the catalog loading (including the CPU bound part) is likely to reap extra
benefits, but as said previously, it’s harder to implement. This is a more modest approach that
still manages to provide a speedup.
–
==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.
The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.