[Geoserver-users] geoserver 2.22.0 "service unavailable"

Branislav_Gacesa · December 1, 2022, 11:51am

Dear users,

I am experiencing an issue while trying to put new geoserver version in to productions.

I have tried both java 8 and 17 running on EC2, rocky enterprise linux 9.1 and ubuntu linux 22.04 environments.

Please find attached error message.

Thanks for your time,

With all the best!

startup_log_v2_20_0_3 (115 KB)

···

Branislav Gaćeša

DevOps Engineer

Engenering&Delivery | HTEC Group Inc

branislav.gacesa@anonymised.com

htecgroup.com

+387 66 808 348

Mladena Stojanovića 30, Banja Luka, BiH

Branislav_Gacesa · December 7, 2022, 1:45pm

Hello geoserver team,

Any experience with subject?

Any information or success story is very helpful.

Thanks!

···

Branislav Gaćeša

DevOps Engineer

Engenering&Delivery | HTEC Group Inc

branislav.gacesa@…11662…

htecgroup.com

+387 66 808 348

Mladena Stojanovića 30, Banja Luka, BiH

jive · December 15, 2022, 3:49pm

Your log seems to be from quite a complicated environment, the final exception is:

2022-11-30 17:02:42.260:DBUG:oeji.ChannelEndPoint:qtp1698097425-21: Ignoring key update for cancelled key SocketChannelEndPoint@anonymised.com{l=/[0:0:0:0:0:0:0:1]:8080,r=/[0:0:0:0:0:0:0:1]:42834,CLOSED,fill=-,flush=-,to=3004/30000}{io=0/0,kio=-1,kro=-1}->HttpConnection@anonymised.com[p=HttpParser{s=CLOSED,0 of -1},g=HttpGenerator@anonymised.com{s=START}]=>HttpChannelOverHttp@anonymised.com1685…{s=HttpChannelState@anonymised.com{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0}
java.nio.channels.CancelledKeyException
at java.base/sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:75)
at java.base/sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:104)
at org.eclipse.jetty.io.ChannelEndPoint.updateKey(ChannelEndPoint.java:379)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.updateKeys(ManagedSelector.java:714)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:833)

Looks to be something in jetty?! Can it actually open the port 8080 in your environment?

···

–
Jody Garnett

Philipp_Letschert · December 19, 2022, 2:06pm

Hello!

similar issue with 2.22.0 in prod here.

Geoserver process stopped/gone for any reason, but left pidfile in place and socket on port 8080 still open.

Unfortunately we don't have additional information yet, but will monitor this.

Possible solution for manual restart:

1. kill geoserver java process

2. remove pidfile /var/run/geoserver.pid

3. check for open socket on Port 8080 with lsof -i :8080 and kill process for this socket

4. start geoserver /etc/init.d/geoserver start

best regards

Phil

Jody Garnett <jody.garnett@anonymised.com> hat am 15. Dezember 2022 16:49 geschrieben:

Your log seems to be from quite a complicated environment, the final exception is:

2022-11-30 17:02:42.260:DBUG:oeji.ChannelEndPoint:qtp1698097425-21: Ignoring key update for cancelled key SocketChannelEndPoint@anonymised.com{l=/[0:0:0:0:0:0:0:1]:8080,r=/[0:0:0:0:0:0:0:1]:42834,CLOSED,fill=-,flush=-,to=3004/30000}{io=0/0,kio=-1,kro=-1}->HttpConnection@anonymised.com[p=HttpParser{s=CLOSED,0 of -1},g=HttpGenerator@anonymised.com{s=START}]=>HttpChannelOverHttp@anonymised.com{s=HttpChannelState@anonymised.com{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0}
java.nio.channels.CancelledKeyException
at java.base/sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:75)
at java.base/sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:104)
at org.eclipse.jetty.io.ChannelEndPoint.updateKey(ChannelEndPoint.java:379)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.updateKeys(ManagedSelector.java:714)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:833)

Looks to be something in jetty?! Can it actually open the port 8080 in your environment?
--
Jody Garnett

On Thu, Dec 1, 2022 at 7:27 AM Branislav Gacesa <branislav.gacesa@anonymised.com2...> wrote:
> Dear users,
> I am experiencing an issue while trying to put new geoserver version in to productions.
>
>
> I have tried both java 8 and 17 running on EC2, rocky enterprise linux 9.1 and ubuntu linux 22.04 environments.
>
>
> Please find attached error message.
> Thanks for your time,
> With all the best!
>
>
>
> https://www.facebook.com/htecgroup/ https://twitter.com/HTECgroup https://www.linkedin.com/company/htecgroup https://www.instagram.com/htecgroup/ Branislav Gaćeša
> DevOps Engineer
> Engenering&Delivery | HTEC Group Inc
>
> branislav.gacesa@anonymised.com
> htecgroup.com (https://htecgroup.com/)
> +387 66 808 348
> Mladena Stojanovića 30, Banja Luka, BiH
>
>
> _______________________________________________
> Geoserver-users mailing list
>
> Please make sure you read the following two resources before posting to this list:
> - Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
> - The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html
>
> If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer
>
>
> Geoserver-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-users
_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer

Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

Branislav_Gacesa · February 10, 2023, 12:32pm

Hello Team,

While trying to import 14+GB big geopkg file I am experiencing java error as shown in attachment.

Please note that peak ram consumption during import is cca 27 out of 32 GB available memmory.

We even played with fine tunning of export JAVA_OPTS=“-Xms4096m -Xmx16384m -XX:MaxPermSize=4096m” but nothing changed.

Any thoughts?

Have a nice weekend…

Thanks.

Branislav Gaćeša

error_import_big_geopkg.txt (16.3 KB)

Hans_Yperman · February 10, 2023, 3:39pm

Some non-expert advise:

Your stack trace basically says that geoserver is extracting 1 tile / PNG file from the geopkg, and it is too big:

at java.desktop/java.awt.image.DataBufferByte.<init>(DataBufferByte.java:92)
at java.desktop/java.awt.image.ComponentSampleModel.createDataBuffer(ComponentSampleModel.java:439)
at java.desktop/java.awt.image.Raster.createWritableRaster(Raster.java:1005)
at java.desktop/javax.imageio.ImageTypeSpecifier.createBufferedImage(ImageTypeSpecifier.java:1074)
at java.desktop/javax.imageio.ImageReader.getDestination(ImageReader.java:2877)
at java.desktop/com.sun.imageio.plugins.png.PNGImageReader.readImage(PNGImageReader.java:1469)

An uncompressed image, being width times height pixels, can be a lot bigger than a compressed PNG, so you might need more than 14Gb. Hence, I’d advise against playing with the memory settings: Even if you manage to import it in geoserver, you’ll be in trouble every time you try to look at it, as it will again use a high amount of memory.

I think you’ll have to extract the geopkg, look at the pixel size of the images, and try to split up images that are too big, while keeping the geopkg metadata intact.

I can’t help you there, unfortunately. My stone age advise would be to go in there with sqlite and extract some blobs, but there are probably better suited geo-specific tools available for creating tiles.

···

Hans Yperman

Department IT

Vlaams Instituut voor de Zee vzw

InnovOcean Campus, Jacobsenstraat 1

8400 Oostende, België

+32 (0) 59 33 61 13

hans.yperman@…1715…

www.vliz.be

Van: Branislav Gacesa <branislav.gacesa@…11662…>
Verzonden: vrijdag 10 februari 2023 13:32
Aan: geoserver-users@lists.sourceforge.net geoserver-users@lists.sourceforge.net
CC: Kosta Cavic <kosta.cavic@…11662…>
Onderwerp: [Geoserver-users] java.lang.OutOfMemoryError: Java heap space