Deadlock Issue for GeoWebCache; GWC-Seed Process

tomo · July 3, 2025, 6:55am

We are facing a deadlock issue when generating tile caches using GeoServer (version 2.19.0) on Linux. Could you kindly provide us a solution for that? (Even some information and/or temporary workaround would be much appreciated)

The details are below:

A Java application sends seed requests to GeoServer.
GWC seed function(*threadCount=1) generates Tiles.
Dividing processing targets of this function into multiple segments, each runs in parallel across multiple peocess.
the value of gutter is up to around 2000 on Metatiling, to avoid splitting long strings
Under the conditions those above, the operation occasionally freeze and a thread-deadlock is deteced (on thread-dump by jstack).

Supplemental Infromation:
The deadlock occurs infrequently, but the frequency increases when the concurrency of seed requests is high.

Found one Java-level deadlock:
=============================
"GWC Seeder Thread-27":
  waiting for ownable synchronizer 0x00007f82789dcc78, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "GWC Seeder Thread-10"
"GWC Seeder Thread-10":
  waiting for ownable synchronizer 0x00007f827697c370, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "GWC Seeder Thread-8"
"GWC Seeder Thread-8":
  waiting for ownable synchronizer 0x00007f827697d4b0, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "GWC Seeder Thread-7"
"GWC Seeder Thread-7":
  waiting for ownable synchronizer 0x00007f827697c370, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "GWC Seeder Thread-8"

Java stack information for the threads listed above:
===================================================
"GWC Seeder Thread-27":
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00007f82789dcc78> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
	at org.geowebcache.locks.MemoryLockProvider.getLock(MemoryLockProvider.java:43)
	at org.geoserver.gwc.ConfigurableLockProvider.getLock(ConfigurableLockProvider.java:24)
	at org.geoserver.gwc.layer.GeoServerTileLayer.getMetatilingReponse(GeoServerTileLayer.java:632)
	at org.geoserver.gwc.layer.GeoServerTileLayer.seedTile(GeoServerTileLayer.java:852)
	at org.geowebcache.seed.SeedTask.doActionInternal(SeedTask.java:140)
	at org.geowebcache.seed.GWCTask.doAction(GWCTask.java:78)
	at org.geowebcache.seed.MTSeeder.call(MTSeeder.java:33)
	at org.geowebcache.seed.MTSeeder.call(MTSeeder.java:22)
	at org.geoserver.gwc.seed.SeederThreadLocalTransferExecutor.lambda$submit$1(SeederThreadLocalTransferExecutor.java:53)
	at org.geoserver.gwc.seed.SeederThreadLocalTransferExecutor$$Lambda$434/1193376748.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
"GWC Seeder Thread-10":
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00007f827697c370> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
	at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
	at org.geowebcache.storage.blobstore.file.LayerMetadataStore.loadLayerMetadata(LayerMetadataStore.java:288)
	at org.geowebcache.storage.blobstore.file.LayerMetadataStore.loadLayerMetadata(LayerMetadataStore.java:337)
	at org.geowebcache.storage.blobstore.file.LayerMetadataStore.writeMetadataOptimisticLock(LayerMetadataStore.java:162)
	at org.geowebcache.storage.blobstore.file.LayerMetadataStore.putEntry(LayerMetadataStore.java:118)
	at org.geowebcache.storage.blobstore.file.FileBlobStore.putLayerMetadata(FileBlobStore.java:657)
	at org.geowebcache.storage.blobstore.file.FileBlobStore.persistParameterMap(FileBlobStore.java:581)
	at org.geowebcache.storage.blobstore.file.FileBlobStore.lambda$put$0(FileBlobStore.java:477)
	at org.geowebcache.storage.blobstore.file.FileBlobStore$$Lambda$570/1563126629.run(Unknown Source)
	at org.geowebcache.storage.blobstore.file.FileBlobStore.getFileHandleTile(FileBlobStore.java:528)
	at org.geowebcache.storage.blobstore.file.FileBlobStore.put(FileBlobStore.java:479)
	at org.geowebcache.storage.CompositeBlobStore.lambda$put$6(CompositeBlobStore.java:186)
	at org.geowebcache.storage.CompositeBlobStore$$Lambda$568/1971368653.run(Unknown Source)
	at org.geowebcache.storage.CompositeBlobStore.lambda$readActionUnsafe$20(CompositeBlobStore.java:508)
	at org.geowebcache.storage.CompositeBlobStore$$Lambda$569/2022073402.get(Unknown Source)
	at org.geowebcache.storage.CompositeBlobStore.readFunctionUnsafe(CompositeBlobStore.java:490)
	at org.geowebcache.storage.CompositeBlobStore.readActionUnsafe(CompositeBlobStore.java:505)
	at org.geowebcache.storage.CompositeBlobStore.put(CompositeBlobStore.java:186)
	at org.geoserver.gwc.ConfigurableBlobStore.put(ConfigurableBlobStore.java:225)
	at org.geowebcache.storage.DefaultStorageBroker.put(DefaultStorageBroker.java:85)
	at org.geowebcache.layer.TileLayer.saveTiles(TileLayer.java:543)
	at org.geoserver.gwc.layer.GeoServerTileLayer.getMetatilingReponse(GeoServerTileLayer.java:654)
	at org.geoserver.gwc.layer.GeoServerTileLayer.seedTile(GeoServerTileLayer.java:852)
	at org.geowebcache.seed.SeedTask.doActionInternal(SeedTask.java:140)
	at org.geowebcache.seed.GWCTask.doAction(GWCTask.java:78)
	at org.geowebcache.seed.MTSeeder.call(MTSeeder.java:33)
	at org.geowebcache.seed.MTSeeder.call(MTSeeder.java:22)
	at org.geoserver.gwc.seed.SeederThreadLocalTransferExecutor.lambda$submit$1(SeederThreadLocalTransferExecutor.java:53)
	at org.geoserver.gwc.seed.SeederThreadLocalTransferExecutor$$Lambda$434/1193376748.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
"GWC Seeder Thread-8":
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00007f827697d4b0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
	at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
	at org.geowebcache.storage.blobstore.file.LayerMetadataStore.writeMetadataFile(LayerMetadataStore.java:250)
	at org.geowebcache.storage.blobstore.file.LayerMetadataStore.writeTempMetadataFile(LayerMetadataStore.java:234)
	at org.geowebcache.storage.blobstore.file.LayerMetadataStore.writeMetadataOptimisticLock(LayerMetadataStore.java:175)
	at org.geowebcache.storage.blobstore.file.LayerMetadataStore.putEntry(LayerMetadataStore.java:118)
	at org.geowebcache.storage.blobstore.file.FileBlobStore.putLayerMetadata(FileBlobStore.java:657)
	at org.geowebcache.storage.blobstore.file.FileBlobStore.persistParameterMap(FileBlobStore.java:581)
	at org.geowebcache.storage.blobstore.file.FileBlobStore.lambda$put$0(FileBlobStore.java:477)
	at org.geowebcache.storage.blobstore.file.FileBlobStore$$Lambda$570/1563126629.run(Unknown Source)
	at org.geowebcache.storage.blobstore.file.FileBlobStore.getFileHandleTile(FileBlobStore.java:528)
	at org.geowebcache.storage.blobstore.file.FileBlobStore.put(FileBlobStore.java:479)
	at org.geowebcache.storage.CompositeBlobStore.lambda$put$6(CompositeBlobStore.java:186)
	at org.geowebcache.storage.CompositeBlobStore$$Lambda$568/1971368653.run(Unknown Source)
	at org.geowebcache.storage.CompositeBlobStore.lambda$readActionUnsafe$20(CompositeBlobStore.java:508)
	at org.geowebcache.storage.CompositeBlobStore$$Lambda$569/2022073402.get(Unknown Source)
	at org.geowebcache.storage.CompositeBlobStore.readFunctionUnsafe(CompositeBlobStore.java:490)
	at org.geowebcache.storage.CompositeBlobStore.readActionUnsafe(CompositeBlobStore.java:505)
	at org.geowebcache.storage.CompositeBlobStore.put(CompositeBlobStore.java:186)
	at org.geoserver.gwc.ConfigurableBlobStore.put(ConfigurableBlobStore.java:225)
	at org.geowebcache.storage.DefaultStorageBroker.put(DefaultStorageBroker.java:85)
	at org.geowebcache.layer.TileLayer.saveTiles(TileLayer.java:543)
	at org.geoserver.gwc.layer.GeoServerTileLayer.getMetatilingReponse(GeoServerTileLayer.java:654)
	at org.geoserver.gwc.layer.GeoServerTileLayer.seedTile(GeoServerTileLayer.java:852)
	at org.geowebcache.seed.SeedTask.doActionInternal(SeedTask.java:140)
	at org.geowebcache.seed.GWCTask.doAction(GWCTask.java:78)
	at org.geowebcache.seed.MTSeeder.call(MTSeeder.java:33)
	at org.geowebcache.seed.MTSeeder.call(MTSeeder.java:22)
	at org.geoserver.gwc.seed.SeederThreadLocalTransferExecutor.lambda$submit$1(SeederThreadLocalTransferExecutor.java:53)
	at org.geoserver.gwc.seed.SeederThreadLocalTransferExecutor$$Lambda$434/1193376748.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
"GWC Seeder Thread-7":
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00007f827697c370> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
	at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
	at org.geowebcache.storage.blobstore.file.LayerMetadataStore.writeMetadataFile(LayerMetadataStore.java:250)
	at org.geowebcache.storage.blobstore.file.LayerMetadataStore.writeTempMetadataFile(LayerMetadataStore.java:234)
	at org.geowebcache.storage.blobstore.file.LayerMetadataStore.writeMetadataOptimisticLock(LayerMetadataStore.java:175)
	at org.geowebcache.storage.blobstore.file.LayerMetadataStore.putEntry(LayerMetadataStore.java:118)
	at org.geowebcache.storage.blobstore.file.FileBlobStore.putLayerMetadata(FileBlobStore.java:657)
	at org.geowebcache.storage.blobstore.file.FileBlobStore.persistParameterMap(FileBlobStore.java:581)
	at org.geowebcache.storage.blobstore.file.FileBlobStore.lambda$put$0(FileBlobStore.java:477)
	at org.geowebcache.storage.blobstore.file.FileBlobStore$$Lambda$570/1563126629.run(Unknown Source)
	at org.geowebcache.storage.blobstore.file.FileBlobStore.getFileHandleTile(FileBlobStore.java:528)
	at org.geowebcache.storage.blobstore.file.FileBlobStore.put(FileBlobStore.java:479)
	at org.geowebcache.storage.CompositeBlobStore.lambda$put$6(CompositeBlobStore.java:186)
	at org.geowebcache.storage.CompositeBlobStore$$Lambda$568/1971368653.run(Unknown Source)
	at org.geowebcache.storage.CompositeBlobStore.lambda$readActionUnsafe$20(CompositeBlobStore.java:508)
	at org.geowebcache.storage.CompositeBlobStore$$Lambda$569/2022073402.get(Unknown Source)
	at org.geowebcache.storage.CompositeBlobStore.readFunctionUnsafe(CompositeBlobStore.java:490)
	at org.geowebcache.storage.CompositeBlobStore.readActionUnsafe(CompositeBlobStore.java:505)
	at org.geowebcache.storage.CompositeBlobStore.put(CompositeBlobStore.java:186)
	at org.geoserver.gwc.ConfigurableBlobStore.put(ConfigurableBlobStore.java:225)
	at org.geowebcache.storage.DefaultStorageBroker.put(DefaultStorageBroker.java:85)
	at org.geowebcache.layer.TileLayer.saveTiles(TileLayer.java:543)
	at org.geoserver.gwc.layer.GeoServerTileLayer.getMetatilingReponse(GeoServerTileLayer.java:654)
	at org.geoserver.gwc.layer.GeoServerTileLayer.seedTile(GeoServerTileLayer.java:852)
	at org.geowebcache.seed.SeedTask.doActionInternal(SeedTask.java:140)
	at org.geowebcache.seed.GWCTask.doAction(GWCTask.java:78)
	at org.geowebcache.seed.MTSeeder.call(MTSeeder.java:33)
	at org.geowebcache.seed.MTSeeder.call(MTSeeder.java:22)
	at org.geoserver.gwc.seed.SeederThreadLocalTransferExecutor.lambda$submit$1(SeederThreadLocalTransferExecutor.java:53)
	at org.geoserver.gwc.seed.SeederThreadLocalTransferExecutor$$Lambda$434/1193376748.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

We would provide more details if neccessaly.
Thank you for your supports always.

aaime-geosolutions · July 3, 2025, 7:19am

GeoServer 2.19.0 is dangerously old (security patches wise) and has been out of support for several years. The seeding process has also been modified in the meantime. Please upgrade to 2.27.1 and try the seeding again, if it still deadlocks then we have something to look into.

Cheers
Andrea

tomo · July 7, 2025, 9:11am

Thank you for your response.
Due to operational constraints of our system, it is difficult to upgrade the version. Therefore, we would like to address the issue without changing the current version.

We have also reviewed past cases in the GeoServer forum and community, but were unable to find any applicable solutions.

aaime-geosolutions · July 7, 2025, 9:30am

Hi,
I’m afraid you’ll be on your own, the GeoServer community can only help with new changes/releases (I don’t remember configurable properties that might help with a deadlock, but if anyone knows one, please speak up).

Commercial support providers may be persuaded to perform custom fixes to a private 2.19.0 fork just for you, see here:
https://geoserver.org/support/

Cheers
Andrea