We are facing a deadlock issue when generating tile caches using GeoServer (version 2.19.0) on Linux. Could you kindly provide us a solution for that? (Even some information and/or temporary workaround would be much appreciated)
The details are below:
- A Java application sends seed requests to GeoServer.
- GWC seed function(*threadCount=1) generates Tiles.
- Dividing processing targets of this function into multiple segments, each runs in parallel across multiple peocess.
- the value of gutter is up to around 2000 on Metatiling, to avoid splitting long strings
- Under the conditions those above, the operation occasionally freeze and a thread-deadlock is deteced (on thread-dump by jstack).
Supplemental Infromation:
The deadlock occurs infrequently, but the frequency increases when the concurrency of seed requests is high.
Found one Java-level deadlock:
=============================
"GWC Seeder Thread-27":
waiting for ownable synchronizer 0x00007f82789dcc78, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "GWC Seeder Thread-10"
"GWC Seeder Thread-10":
waiting for ownable synchronizer 0x00007f827697c370, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
which is held by "GWC Seeder Thread-8"
"GWC Seeder Thread-8":
waiting for ownable synchronizer 0x00007f827697d4b0, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
which is held by "GWC Seeder Thread-7"
"GWC Seeder Thread-7":
waiting for ownable synchronizer 0x00007f827697c370, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
which is held by "GWC Seeder Thread-8"
Java stack information for the threads listed above:
===================================================
"GWC Seeder Thread-27":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007f82789dcc78> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at org.geowebcache.locks.MemoryLockProvider.getLock(MemoryLockProvider.java:43)
at org.geoserver.gwc.ConfigurableLockProvider.getLock(ConfigurableLockProvider.java:24)
at org.geoserver.gwc.layer.GeoServerTileLayer.getMetatilingReponse(GeoServerTileLayer.java:632)
at org.geoserver.gwc.layer.GeoServerTileLayer.seedTile(GeoServerTileLayer.java:852)
at org.geowebcache.seed.SeedTask.doActionInternal(SeedTask.java:140)
at org.geowebcache.seed.GWCTask.doAction(GWCTask.java:78)
at org.geowebcache.seed.MTSeeder.call(MTSeeder.java:33)
at org.geowebcache.seed.MTSeeder.call(MTSeeder.java:22)
at org.geoserver.gwc.seed.SeederThreadLocalTransferExecutor.lambda$submit$1(SeederThreadLocalTransferExecutor.java:53)
at org.geoserver.gwc.seed.SeederThreadLocalTransferExecutor$$Lambda$434/1193376748.call(Unknown Source)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
"GWC Seeder Thread-10":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007f827697c370> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at org.geowebcache.storage.blobstore.file.LayerMetadataStore.loadLayerMetadata(LayerMetadataStore.java:288)
at org.geowebcache.storage.blobstore.file.LayerMetadataStore.loadLayerMetadata(LayerMetadataStore.java:337)
at org.geowebcache.storage.blobstore.file.LayerMetadataStore.writeMetadataOptimisticLock(LayerMetadataStore.java:162)
at org.geowebcache.storage.blobstore.file.LayerMetadataStore.putEntry(LayerMetadataStore.java:118)
at org.geowebcache.storage.blobstore.file.FileBlobStore.putLayerMetadata(FileBlobStore.java:657)
at org.geowebcache.storage.blobstore.file.FileBlobStore.persistParameterMap(FileBlobStore.java:581)
at org.geowebcache.storage.blobstore.file.FileBlobStore.lambda$put$0(FileBlobStore.java:477)
at org.geowebcache.storage.blobstore.file.FileBlobStore$$Lambda$570/1563126629.run(Unknown Source)
at org.geowebcache.storage.blobstore.file.FileBlobStore.getFileHandleTile(FileBlobStore.java:528)
at org.geowebcache.storage.blobstore.file.FileBlobStore.put(FileBlobStore.java:479)
at org.geowebcache.storage.CompositeBlobStore.lambda$put$6(CompositeBlobStore.java:186)
at org.geowebcache.storage.CompositeBlobStore$$Lambda$568/1971368653.run(Unknown Source)
at org.geowebcache.storage.CompositeBlobStore.lambda$readActionUnsafe$20(CompositeBlobStore.java:508)
at org.geowebcache.storage.CompositeBlobStore$$Lambda$569/2022073402.get(Unknown Source)
at org.geowebcache.storage.CompositeBlobStore.readFunctionUnsafe(CompositeBlobStore.java:490)
at org.geowebcache.storage.CompositeBlobStore.readActionUnsafe(CompositeBlobStore.java:505)
at org.geowebcache.storage.CompositeBlobStore.put(CompositeBlobStore.java:186)
at org.geoserver.gwc.ConfigurableBlobStore.put(ConfigurableBlobStore.java:225)
at org.geowebcache.storage.DefaultStorageBroker.put(DefaultStorageBroker.java:85)
at org.geowebcache.layer.TileLayer.saveTiles(TileLayer.java:543)
at org.geoserver.gwc.layer.GeoServerTileLayer.getMetatilingReponse(GeoServerTileLayer.java:654)
at org.geoserver.gwc.layer.GeoServerTileLayer.seedTile(GeoServerTileLayer.java:852)
at org.geowebcache.seed.SeedTask.doActionInternal(SeedTask.java:140)
at org.geowebcache.seed.GWCTask.doAction(GWCTask.java:78)
at org.geowebcache.seed.MTSeeder.call(MTSeeder.java:33)
at org.geowebcache.seed.MTSeeder.call(MTSeeder.java:22)
at org.geoserver.gwc.seed.SeederThreadLocalTransferExecutor.lambda$submit$1(SeederThreadLocalTransferExecutor.java:53)
at org.geoserver.gwc.seed.SeederThreadLocalTransferExecutor$$Lambda$434/1193376748.call(Unknown Source)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
"GWC Seeder Thread-8":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007f827697d4b0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
at org.geowebcache.storage.blobstore.file.LayerMetadataStore.writeMetadataFile(LayerMetadataStore.java:250)
at org.geowebcache.storage.blobstore.file.LayerMetadataStore.writeTempMetadataFile(LayerMetadataStore.java:234)
at org.geowebcache.storage.blobstore.file.LayerMetadataStore.writeMetadataOptimisticLock(LayerMetadataStore.java:175)
at org.geowebcache.storage.blobstore.file.LayerMetadataStore.putEntry(LayerMetadataStore.java:118)
at org.geowebcache.storage.blobstore.file.FileBlobStore.putLayerMetadata(FileBlobStore.java:657)
at org.geowebcache.storage.blobstore.file.FileBlobStore.persistParameterMap(FileBlobStore.java:581)
at org.geowebcache.storage.blobstore.file.FileBlobStore.lambda$put$0(FileBlobStore.java:477)
at org.geowebcache.storage.blobstore.file.FileBlobStore$$Lambda$570/1563126629.run(Unknown Source)
at org.geowebcache.storage.blobstore.file.FileBlobStore.getFileHandleTile(FileBlobStore.java:528)
at org.geowebcache.storage.blobstore.file.FileBlobStore.put(FileBlobStore.java:479)
at org.geowebcache.storage.CompositeBlobStore.lambda$put$6(CompositeBlobStore.java:186)
at org.geowebcache.storage.CompositeBlobStore$$Lambda$568/1971368653.run(Unknown Source)
at org.geowebcache.storage.CompositeBlobStore.lambda$readActionUnsafe$20(CompositeBlobStore.java:508)
at org.geowebcache.storage.CompositeBlobStore$$Lambda$569/2022073402.get(Unknown Source)
at org.geowebcache.storage.CompositeBlobStore.readFunctionUnsafe(CompositeBlobStore.java:490)
at org.geowebcache.storage.CompositeBlobStore.readActionUnsafe(CompositeBlobStore.java:505)
at org.geowebcache.storage.CompositeBlobStore.put(CompositeBlobStore.java:186)
at org.geoserver.gwc.ConfigurableBlobStore.put(ConfigurableBlobStore.java:225)
at org.geowebcache.storage.DefaultStorageBroker.put(DefaultStorageBroker.java:85)
at org.geowebcache.layer.TileLayer.saveTiles(TileLayer.java:543)
at org.geoserver.gwc.layer.GeoServerTileLayer.getMetatilingReponse(GeoServerTileLayer.java:654)
at org.geoserver.gwc.layer.GeoServerTileLayer.seedTile(GeoServerTileLayer.java:852)
at org.geowebcache.seed.SeedTask.doActionInternal(SeedTask.java:140)
at org.geowebcache.seed.GWCTask.doAction(GWCTask.java:78)
at org.geowebcache.seed.MTSeeder.call(MTSeeder.java:33)
at org.geowebcache.seed.MTSeeder.call(MTSeeder.java:22)
at org.geoserver.gwc.seed.SeederThreadLocalTransferExecutor.lambda$submit$1(SeederThreadLocalTransferExecutor.java:53)
at org.geoserver.gwc.seed.SeederThreadLocalTransferExecutor$$Lambda$434/1193376748.call(Unknown Source)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
"GWC Seeder Thread-7":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007f827697c370> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
at org.geowebcache.storage.blobstore.file.LayerMetadataStore.writeMetadataFile(LayerMetadataStore.java:250)
at org.geowebcache.storage.blobstore.file.LayerMetadataStore.writeTempMetadataFile(LayerMetadataStore.java:234)
at org.geowebcache.storage.blobstore.file.LayerMetadataStore.writeMetadataOptimisticLock(LayerMetadataStore.java:175)
at org.geowebcache.storage.blobstore.file.LayerMetadataStore.putEntry(LayerMetadataStore.java:118)
at org.geowebcache.storage.blobstore.file.FileBlobStore.putLayerMetadata(FileBlobStore.java:657)
at org.geowebcache.storage.blobstore.file.FileBlobStore.persistParameterMap(FileBlobStore.java:581)
at org.geowebcache.storage.blobstore.file.FileBlobStore.lambda$put$0(FileBlobStore.java:477)
at org.geowebcache.storage.blobstore.file.FileBlobStore$$Lambda$570/1563126629.run(Unknown Source)
at org.geowebcache.storage.blobstore.file.FileBlobStore.getFileHandleTile(FileBlobStore.java:528)
at org.geowebcache.storage.blobstore.file.FileBlobStore.put(FileBlobStore.java:479)
at org.geowebcache.storage.CompositeBlobStore.lambda$put$6(CompositeBlobStore.java:186)
at org.geowebcache.storage.CompositeBlobStore$$Lambda$568/1971368653.run(Unknown Source)
at org.geowebcache.storage.CompositeBlobStore.lambda$readActionUnsafe$20(CompositeBlobStore.java:508)
at org.geowebcache.storage.CompositeBlobStore$$Lambda$569/2022073402.get(Unknown Source)
at org.geowebcache.storage.CompositeBlobStore.readFunctionUnsafe(CompositeBlobStore.java:490)
at org.geowebcache.storage.CompositeBlobStore.readActionUnsafe(CompositeBlobStore.java:505)
at org.geowebcache.storage.CompositeBlobStore.put(CompositeBlobStore.java:186)
at org.geoserver.gwc.ConfigurableBlobStore.put(ConfigurableBlobStore.java:225)
at org.geowebcache.storage.DefaultStorageBroker.put(DefaultStorageBroker.java:85)
at org.geowebcache.layer.TileLayer.saveTiles(TileLayer.java:543)
at org.geoserver.gwc.layer.GeoServerTileLayer.getMetatilingReponse(GeoServerTileLayer.java:654)
at org.geoserver.gwc.layer.GeoServerTileLayer.seedTile(GeoServerTileLayer.java:852)
at org.geowebcache.seed.SeedTask.doActionInternal(SeedTask.java:140)
at org.geowebcache.seed.GWCTask.doAction(GWCTask.java:78)
at org.geowebcache.seed.MTSeeder.call(MTSeeder.java:33)
at org.geowebcache.seed.MTSeeder.call(MTSeeder.java:22)
at org.geoserver.gwc.seed.SeederThreadLocalTransferExecutor.lambda$submit$1(SeederThreadLocalTransferExecutor.java:53)
at org.geoserver.gwc.seed.SeederThreadLocalTransferExecutor$$Lambda$434/1193376748.call(Unknown Source)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
We would provide more details if neccessaly.
Thank you for your supports always.