[Geoserver-users] Geoserver crashes

Dear all

I’m having some trouble with geoserver 1.7.4.
I’m using it on three servers all with the same stack:

MS Windows server 2003 Enterprise edition 32 bit
Sun JDK 1.6.0_11-b03
Apache Tomcat 6.0.18
Oracle client 10.2.0.3

Without any correlation with some specific WMS or WFS request the jvm crashes and tomcat service restart.

I can found an hh_xxxx.pid file and a sqlnet.log in the main folder of Tomcat. I didn’t find anything relevant on geoserver.log.

The sqlnet.log is filled with error messages like this


Fatal NI connect error 12560, connecting to:
(DESCRIPTION=(ADDRESS=(PROTOCOL=BEQ)(PROGRAM=oracle)(ARGV0=oracleORCL)(ARGS=‘(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))’))(CONNECT_DATA=(SID=ORCL)(CID=(PROGRAM=C:\opt\Tomcat6018\bin\tomcat6.exe)(HOST=SITR-MAP-03)(USER=SYSTEM))))

VERSION INFORMATION:
TNS for 32-bit Windows: Version 10.2.0.3.0 - Production
Oracle Bequeath NT Protocol Adapter for 32-bit Windows: Version 10.2.0.3.0 - Production
Windows NT TCP/IP NT Protocol Adapter for 32-bit Windows: Version 10.2.0.3.0 - Production
Time: 22-MAY-2009 09:18:12
Tracing not turned on.
Tns error struct:
ns main err code: 12560
TNS-12560: TNS:protocol adapter error
ns secondary err code: 0
nt main err code: 530
TNS-00530: Protocol adapter error
nt secondary err code: 126
nt OS err code: 0

I’m pasting the content of one of the hh_xxxx.pid log file. Is there any way to understand what exactly is

Thank you for any hint on how to debug this

Stefano Iacovella

An unexpected error has been detected by Java Runtime Environment:

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6d204aa6, pid=1528, tid=1816

Java VM: Java HotSpot™ Server VM (11.0-b16 mixed mode windows-x86)

Problematic frame:

C [tcnative-1.dll+0x4aa6]

If you would like to submit a bug report, please visit:

http://java.sun.com/webapps/bugreport/crash.jsp

The crash happened outside the Java Virtual Machine in native code.

See problematic frame for where to report the bug.

--------------- T H R E A D ---------------

Current thread (0x6bee2400): JavaThread “Finalizer” daemon [_thread_in_native, id=1816, stack(0x6c370000,0x6c3c0000)]

siginfo: ExceptionCode=0xc0000005, reading address 0x00000020

Registers:
EAX=0x00000000, EBX=0x0000000e, ECX=0x00000000, EDX=0x00000000
ESP=0x6c3bf610, EBP=0x6c3bf628, ESI=0x00000000, EDI=0x71389788
EIP=0x6d204aa6, EFLAGS=0x00010246

Top of Stack: (sp=0x6c3bf610)
0x6c3bf610: 00000000 00000000 6c3bf630 6bee2400
0x6c3bf620: 042cdb00 042cdb00 6c3bf66c 00d3a27e
0x6c3bf630: 0000000e 6c3bf674 71389788 00000000
0x6c3bf640: 00000000 0000000e 05dfc430 6c3bf64c
0x6c3bf650: 00000000 6c3bf688 042ce7b8 00000000
0x6c3bf660: 042cdb00 00000000 6c3bf67c 6c3bf6ac
0x6c3bf670: 00d32f85 042ce758 00d38299 0000000e
0x6c3bf680: 00000000 71389788 00000000 6c3bf68c

Instructions: (pc=0x6d204aa6)
0x6d204a96: 55 18 51 8d 0c 16 8b 57 10 03 ca 8b 57 0c 51 52
0x6d204aa6: ff 50 20 85 c0 75 14 03 75 08 3b f3 72 d4 85 c0

Stack: [0x6c370000,0x6c3c0000], sp=0x6c3bf610, free space=317k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [tcnative-1.dll+0x4aa6]
j org.apache.tomcat.jni.Socket.sendbb(JII)I+0
j org.apache.coyote.ajp.AjpAprProcessor.flush()V+22
j org.apache.coyote.ajp.AjpAprProcessor.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V+66
j org.apache.coyote.Response.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V+31
j org.apache.catalina.connector.OutputBuffer.doFlush(Z)V+69
j org.apache.catalina.connector.OutputBuffer.flush()V+2
j org.apache.catalina.connector.CoyoteOutputStream.flush()V+4
j org.geoserver.filters.AlternativesResponseStream.flush()V+11
j java.io.BufferedOutputStream.flush()V+8
j org.geoserver.ows.DispatcherOutputStream.flush()V+4
j javax.imageio.stream.MemoryCacheImageOutputStream.flushBefore(J)V+47
j javax.imageio.stream.MemoryCacheImageOutputStream.close()V+15
j javax.imageio.stream.ImageInputStreamImpl.finalize()V+8
v ~StubRoutines::call_stub

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j org.apache.tomcat.jni.Socket.sendbb(JII)I+0
j org.apache.coyote.ajp.AjpAprProcessor.flush()V+22
j org.apache.coyote.ajp.AjpAprProcessor.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V+66
j org.apache.coyote.Response.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V+31
j org.apache.catalina.connector.OutputBuffer.doFlush(Z)V+69
j org.apache.catalina.connector.OutputBuffer.flush()V+2
j org.apache.catalina.connector.CoyoteOutputStream.flush()V+4
j org.geoserver.filters.AlternativesResponseStream.flush()V+11
j java.io.BufferedOutputStream.flush()V+8
j org.geoserver.ows.DispatcherOutputStream.flush()V+4
j javax.imageio.stream.MemoryCacheImageOutputStream.flushBefore(J)V+47
j javax.imageio.stream.MemoryCacheImageOutputStream.close()V+15
j javax.imageio.stream.ImageInputStreamImpl.finalize()V+8
v ~StubRoutines::call_stub
j java.lang.ref.Finalizer.invokeFinalizeMethod(Ljava/lang/Object;)V+0
j java.lang.ref.Finalizer.runFinalizer()V+45
J java.lang.ref.Finalizer$FinalizerThread.run()V
v ~StubRoutines::call_stub

--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
0x747f5c00 JavaThread “ajp-8009-9” daemon [_thread_in_native, id=620, stack(0x72bb0000,0x72c00000)]
0x7e80ac00 JavaThread “ajp-8009-8” daemon [_thread_blocked, id=204, stack(0x72b60000,0x72bb0000)]
0x6e4af000 JavaThread “ajp-8009-7” daemon [_thread_blocked, id=3368, stack(0x72b10000,0x72b60000)]
0x746ac400 JavaThread “ajp-8009-6” daemon [_thread_blocked, id=1980, stack(0x72ac0000,0x72b10000)]
0x6e82f400 JavaThread “ajp-8009-5” daemon [_thread_blocked, id=3660, stack(0x6d790000,0x6d7e0000)]
0x6c8df800 JavaThread “ajp-8009-4” daemon [_thread_blocked, id=144, stack(0x71d40000,0x71d90000)]
0x73838800 JavaThread “ajp-8009-3” daemon [_thread_blocked, id=4084, stack(0x71ca0000,0x71cf0000)]
0x6e175c00 JavaThread “AWT-Windows” daemon [_thread_in_native, id=4028, stack(0x71cf0000,0x71d40000)]
0x6c6c3c00 JavaThread “Java2D Disposer” daemon [_thread_blocked, id=4020, stack(0x71a90000,0x71ae0000)]
0x6e4eac00 JavaThread “ajp-8009-2” daemon [_thread_blocked, id=4016, stack(0x719e0000,0x71a30000)]
0x733e6400 JavaThread “ajp-8009-1” daemon [_thread_blocked, id=4012, stack(0x71990000,0x719e0000)]
0x7335ec00 JavaThread “ajp-8009-Acceptor-0” daemon [_thread_in_native, id=3228, stack(0x71940000,0x71990000)]
0x6cf87800 JavaThread “ajp-8009-CometPoller-7” daemon [_thread_blocked, id=3224, stack(0x718f0000,0x71940000)]
0x6c996c00 JavaThread “ajp-8009-CometPoller-6” daemon [_thread_blocked, id=3220, stack(0x718a0000,0x718f0000)]
0x73915400 JavaThread “ajp-8009-CometPoller-5” daemon [_thread_blocked, id=3216, stack(0x71850000,0x718a0000)]
0x6df58800 JavaThread “ajp-8009-CometPoller-4” daemon [_thread_blocked, id=3212, stack(0x71800000,0x71850000)]
0x6cabc400 JavaThread “ajp-8009-CometPoller-3” daemon [_thread_blocked, id=3208, stack(0x717b0000,0x71800000)]
0x6c21f800 JavaThread “ajp-8009-CometPoller-2” daemon [_thread_blocked, id=3204, stack(0x71760000,0x717b0000)]
0x7348dc00 JavaThread “ajp-8009-CometPoller-1” daemon [_thread_blocked, id=3200, stack(0x71710000,0x71760000)]
0x73473c00 JavaThread “ajp-8009-CometPoller-0” daemon [_thread_blocked, id=3196, stack(0x716c0000,0x71710000)]
0x6c9f0400 JavaThread “ajp-8009-Poller-7” daemon [_thread_in_native, id=3192, stack(0x71670000,0x716c0000)]
0x737c0400 JavaThread “ajp-8009-Poller-6” daemon [_thread_in_native, id=3188, stack(0x71620000,0x71670000)]
0x6ea76400 JavaThread “ajp-8009-Poller-5” daemon [_thread_in_native, id=3184, stack(0x715d0000,0x71620000)]
0x6cab4c00 JavaThread “ajp-8009-Poller-4” daemon [_thread_in_native, id=3180, stack(0x71180000,0x711d0000)]
0x6cbbc000 JavaThread “ajp-8009-Poller-3” daemon [_thread_in_native, id=3176, stack(0x71130000,0x71180000)]
0x6c18b400 JavaThread “ajp-8009-Poller-2” daemon [_thread_in_native, id=3172, stack(0x710e0000,0x71130000)]
0x6ceba800 JavaThread “ajp-8009-Poller-1” daemon [_thread_blocked, id=3168, stack(0x71090000,0x710e0000)]
0x6caf0400 JavaThread “ajp-8009-Poller-0” daemon [_thread_in_native, id=3164, stack(0x71040000,0x71090000)]
0x6cfe2800 JavaThread “http-8080-Acceptor-0” daemon [_thread_in_native, id=3160, stack(0x70ff0000,0x71040000)]
0x739ad800 JavaThread “http-8080-Sendfile-0” daemon [_thread_blocked, id=3156, stack(0x70fa0000,0x70ff0000)]
0x73935800 JavaThread “http-8080-CometPoller-7” daemon [_thread_blocked, id=3152, stack(0x70f50000,0x70fa0000)]
0x7319c800 JavaThread “http-8080-CometPoller-6” daemon [_thread_blocked, id=3148, stack(0x70f00000,0x70f50000)]
0x6ca20800 JavaThread “http-8080-CometPoller-5” daemon [_thread_blocked, id=3144, stack(0x70eb0000,0x70f00000)]
0x6ca18400 JavaThread “http-8080-CometPoller-4” daemon [_thread_blocked, id=3140, stack(0x70e60000,0x70eb0000)]
0x6ccb6400 JavaThread “http-8080-CometPoller-3” daemon [_thread_blocked, id=3136, stack(0x70e10000,0x70e60000)]
0x6e0f5800 JavaThread “http-8080-CometPoller-2” daemon [_thread_blocked, id=3132, stack(0x70dc0000,0x70e10000)]
0x6cb65000 JavaThread “http-8080-CometPoller-1” daemon [_thread_blocked, id=3128, stack(0x70d70000,0x70dc0000)]
0x6c604000 JavaThread “http-8080-CometPoller-0” daemon [_thread_blocked, id=3124, stack(0x70d20000,0x70d70000)]
0x733d6800 JavaThread “http-8080-Poller-7” daemon [_thread_blocked, id=3120, stack(0x70cd0000,0x70d20000)]
0x6c705800 JavaThread “http-8080-Poller-6” daemon [_thread_blocked, id=3116, stack(0x70c80000,0x70cd0000)]
0x6ce48400 JavaThread “http-8080-Poller-5” daemon [_thread_blocked, id=3112, stack(0x70a30000,0x70a80000)]
0x6ccbe400 JavaThread “http-8080-Poller-4” daemon [_thread_blocked, id=3108, stack(0x709e0000,0x70a30000)]
0x6e0fd400 JavaThread “http-8080-Poller-3” daemon [_thread_blocked, id=3104, stack(0x70990000,0x709e0000)]
0x6c996000 JavaThread “http-8080-Poller-2” daemon [_thread_blocked, id=3100, stack(0x70940000,0x70990000)]
0x6c79a400 JavaThread “http-8080-Poller-1” daemon [_thread_blocked, id=3096, stack(0x708f0000,0x70940000)]
0x6ca10c00 JavaThread “http-8080-Poller-0” daemon [_thread_blocked, id=3092, stack(0x708a0000,0x708f0000)]
0x6ea6ec00 JavaThread “ContainerBackgroundProcessor[StandardEngine[Catalina]]” daemon [_thread_blocked, id=3088, stack(0x70800000,0x70850000)]
0x6ec79400 JavaThread “wcs111timerFactory” [_thread_blocked, id=3084, stack(0x70850000,0x708a0000)]
0x6cf08c00 JavaThread “Timer-1” daemon [_thread_blocked, id=2816, stack(0x70780000,0x707d0000)]
0x6e2b1c00 JavaThread “Timer-0” daemon [_thread_blocked, id=2812, stack(0x6dad0000,0x6db20000)]
0x6ca4bc00 JavaThread “WeakCollectionCleaner” daemon [_thread_blocked, id=2740, stack(0x6d7e0000,0x6d830000)]
0x6c1de400 JavaThread “Thread-1” [_thread_in_native, id=1904, stack(0x6c5a0000,0x6c5f0000)]
0x6c103400 JavaThread “Low Memory Detector” daemon [_thread_blocked, id=1876, stack(0x6c500000,0x6c550000)]
0x6c100800 JavaThread “CompilerThread1” daemon [_thread_blocked, id=1868, stack(0x6c4b0000,0x6c500000)]
0x6c0fb000 JavaThread “CompilerThread0” daemon [_thread_blocked, id=1860, stack(0x6c460000,0x6c4b0000)]
0x6c0f9800 JavaThread “Attach Listener” daemon [_thread_blocked, id=1852, stack(0x6c410000,0x6c460000)]
0x6beef000 JavaThread “Signal Dispatcher” daemon [_thread_blocked, id=1848, stack(0x6c3c0000,0x6c410000)]
=>0x6bee2400 JavaThread “Finalizer” daemon [_thread_in_native, id=1816, stack(0x6c370000,0x6c3c0000)]
0x6bee1000 JavaThread “Reference Handler” daemon [_thread_blocked, id=1812, stack(0x6c320000,0x6c370000)]
0x005f6c00 JavaThread “main” [_thread_in_native, id=1572, stack(0x00550000,0x005a0000)]

Other Threads:
0x6c0f1000 VMThread [stack: 0x6c2d0000,0x6c320000] [id=1804]
0x6c105000 WatcherThread [stack: 0x6c550000,0x6c5a0000] [id=1880]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
PSYoungGen total 137984K, used 1918K [0x60b70000, 0x6b870000, 0x6bd30000)
eden space 97536K, 1% used [0x60b70000,0x60d0d878,0x66ab0000)
from space 40448K, 0% used [0x690f0000,0x691320b0,0x6b870000)
to space 39168K, 0% used [0x66ab0000,0x66ab0000,0x690f0000)
PSOldGen total 146048K, used 64210K [0x07d30000, 0x10bd0000, 0x60b70000)
object space 146048K, 43% used [0x07d30000,0x0bbe4b38,0x10bd0000)
PSPermGen total 63232K, used 47192K [0x03d30000, 0x07af0000, 0x07d30000)
object space 63232K, 74% used [0x03d30000,0x06b46010,0x07af0000)

Dynamic libraries:
0x00400000 - 0x0040f000 C:\opt\Tomcat6018\bin\tomcat6.exe
0x7c800000 - 0x7c8c0000 C:\WINDOWS\system32\ntdll.dll
0x77e40000 - 0x77f42000 C:\WINDOWS\system32\kernel32.dll
0x77380000 - 0x77411000 C:\WINDOWS\system32\USER32.dll
0x77c00000 - 0x77c48000 C:\WINDOWS\system32\GDI32.dll
0x77f50000 - 0x77feb000 C:\WINDOWS\system32\ADVAPI32.dll
0x77c50000 - 0x77cef000 C:\WINDOWS\system32\RPCRT4.dll
0x76f50000 - 0x76f63000 C:\WINDOWS\system32\Secur32.dll
0x77ba0000 - 0x77bfa000 C:\WINDOWS\system32\MSVCRT.dll
0x7c8d0000 - 0x7d0cf000 C:\WINDOWS\system32\SHELL32.dll
0x77da0000 - 0x77df2000 C:\WINDOWS\system32\SHLWAPI.dll
0x77420000 - 0x77523000 C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.3790.3959_x-ww_D8713E55\comctl32.dll
0x6db30000 - 0x6def4000 C:\opt\jdk1.6.0_11\jre\bin\server\jvm.dll
0x76aa0000 - 0x76acd000 C:\WINDOWS\system32\WINMM.dll
0x7c340000 - 0x7c396000 C:\opt\oracle\product\10.2.0\client_1\bin\MSVCR71.dll
0x6d330000 - 0x6d338000 C:\opt\jdk1.6.0_11\jre\bin\hpi.dll
0x76b70000 - 0x76b7b000 C:\WINDOWS\system32\PSAPI.DLL
0x6d860000 - 0x6d86c000 C:\opt\jdk1.6.0_11\jre\bin\verify.dll
0x6d3d0000 - 0x6d3ef000 C:\opt\jdk1.6.0_11\jre\bin\java.dll
0x6d8a0000 - 0x6d8af000 C:\opt\jdk1.6.0_11\jre\bin\zip.dll
0x6d200000 - 0x6d2d2000 C:\opt\Tomcat6018\bin\tcnative-1.dll
0x71c00000 - 0x71c17000 C:\WINDOWS\system32\WS2_32.dll
0x71bf0000 - 0x71bf8000 C:\WINDOWS\system32\WS2HELP.dll
0x71b20000 - 0x71b61000 C:\WINDOWS\system32\MSWSOCK.dll
0x6d340000 - 0x6d375000 C:\WINDOWS\system32\rsaenh.dll
0x6d4f0000 - 0x6d54a000 C:\WINDOWS\system32\hnetcfg.dll
0x71ae0000 - 0x71ae8000 C:\WINDOWS\System32\wshtcpip.dll
0x6d550000 - 0x6d688000 C:\opt\jdk1.6.0_11\jre\bin\awt.dll
0x73070000 - 0x73097000 C:\WINDOWS\system32\WINSPOOL.DRV
0x76290000 - 0x762ad000 C:\WINDOWS\system32\IMM32.dll
0x77670000 - 0x777a9000 C:\WINDOWS\system32\ole32.dll
0x77530000 - 0x775c7000 C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_5.82.3790.3959_x-ww_78FCF8D0\comctl32.dll
0x6d830000 - 0x6d843000 C:\opt\oracle\product\10.2.0\client_1\BIN\ocijdbc10.dll
0x6d9b0000 - 0x6da09000 C:\opt\oracle\product\10.2.0\client_1\bin\OCI.dll
0x6ef00000 - 0x6f156000 C:\opt\oracle\product\10.2.0\client_1\bin\OraClient10.Dll
0x6f160000 - 0x6f247000 C:\opt\oracle\product\10.2.0\client_1\bin\oracore10.dll
0x6f250000 - 0x6f316000 C:\opt\oracle\product\10.2.0\client_1\bin\oranls10.dll
0x6d870000 - 0x6d888000 C:\opt\oracle\product\10.2.0\client_1\bin\oraunls10.dll
0x6d850000 - 0x6d857000 C:\opt\oracle\product\10.2.0\client_1\bin\orauts.dll
0x6d890000 - 0x6d896000 C:\opt\oracle\product\10.2.0\client_1\bin\oravsn10.dll
0x6f320000 - 0x6f413000 C:\opt\oracle\product\10.2.0\client_1\bin\oracommon10.dll
0x6f420000 - 0x6f98b000 C:\opt\oracle\product\10.2.0\client_1\bin\orageneric10.dll
0x6da10000 - 0x6da37000 C:\opt\oracle\product\10.2.0\client_1\bin\orasnls10.dll
0x6f990000 - 0x6fb06000 C:\opt\oracle\product\10.2.0\client_1\bin\oraxml10.dll
0x6da40000 - 0x6da50000 C:\WINDOWS\system32\MSVCIRT.dll
0x6fb10000 - 0x6fc22000 C:\opt\oracle\product\10.2.0\client_1\bin\oran10.dll
0x6da50000 - 0x6da8d000 C:\opt\oracle\product\10.2.0\client_1\bin\oranl10.dll
0x6da90000 - 0x6daa7000 C:\opt\oracle\product\10.2.0\client_1\bin\oranldap10.dll
0x6fc30000 - 0x6fd3c000 C:\opt\oracle\product\10.2.0\client_1\bin\orannzsbb10.dll
0x6fd40000 - 0x6fd93000 C:\opt\oracle\product\10.2.0\client_1\bin\oraldapclnt10.dll
0x6dab0000 - 0x6dacb000 C:\opt\oracle\product\10.2.0\client_1\bin\orancrypt10.dll
0x71bb0000 - 0x71bb9000 C:\WINDOWS\system32\WSOCK32.dll
0x77d00000 - 0x77d8b000 C:\WINDOWS\system32\OLEAUT32.dll
0x6fda0000 - 0x6fded000 C:\opt\oracle\product\10.2.0\client_1\bin\oranro10.dll
0x6db20000 - 0x6db27000 C:\opt\oracle\product\10.2.0\client_1\bin\oranhost10.dll
0x6fdf0000 - 0x6fdf6000 C:\opt\oracle\product\10.2.0\client_1\bin\orancds10.dll
0x6fe00000 - 0x6fe08000 C:\opt\oracle\product\10.2.0\client_1\bin\orantns10.dll
0x6fe10000 - 0x70176000 C:\opt\oracle\product\10.2.0\client_1\bin\orapls10.dll
0x70180000 - 0x70189000 C:\opt\oracle\product\10.2.0\client_1\bin\oraslax10.dll
0x70190000 - 0x70394000 C:\opt\oracle\product\10.2.0\client_1\bin\oraplp10.dll
0x703a0000 - 0x70438000 C:\opt\oracle\product\10.2.0\client_1\bin\orahasgen10.dll
0x70440000 - 0x704af000 C:\opt\oracle\product\10.2.0\client_1\bin\oraocr10.dll
0x704b0000 - 0x704f6000 C:\opt\oracle\product\10.2.0\client_1\bin\oraocrb10.dll
0x71c40000 - 0x71c97000 C:\WINDOWS\system32\NETAPI32.dll
0x70500000 - 0x70510000 C:\opt\oracle\product\10.2.0\client_1\bin\orantcp10.dll
0x70510000 - 0x705aa000 C:\opt\oracle\product\10.2.0\client_1\bin\orasql10.dll
0x76ed0000 - 0x76efa000 C:\WINDOWS\system32\DNSAPI.dll
0x76f70000 - 0x76f77000 C:\WINDOWS\System32\winrnr.dll
0x76f10000 - 0x76f3e000 C:\WINDOWS\system32\WLDAP32.dll
0x76f80000 - 0x76f85000 C:\WINDOWS\system32\rasadhlp.dll
0x707d0000 - 0x707e3000 C:\opt\jdk1.6.0_11\jre\bin\net.dll
0x707f0000 - 0x707f9000 C:\opt\jdk1.6.0_11\jre\bin\nio.dll
0x71a30000 - 0x71a84000 C:\opt\jdk1.6.0_11\jre\bin\fontmanager.dll
0x71af0000 - 0x71b13000 C:\opt\jdk1.6.0_11\jre\bin\dcpr.dll

VM Arguments:
jvm_args: -Dcatalina.home=C:\opt\Tomcat6018 -Dcatalina.base=C:\opt\Tomcat6018 -Djava.endorsed.dirs=C:\opt\Tomcat6018\endorsed -Djava.io.tmpdir=C:\opt\Tomcat6018\temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=C:\opt\Tomcat6018\conf\logging.properties vfprintf -Xms32m -Xmx1600m
java_command:
Launcher Type: generic

Environment Variables:
JAVA_HOME=C:\opt\jdk1.6.0_11
PATH=C:\opt\oracle\product\10.2.0\client_1\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\Intel\DMIX;C:\opt\jdk1.6.0_11\bin;C:\Program Files\NetApp\Windows Host Utilities
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 6 Model 23 Stepping 6, GenuineIntel

--------------- S Y S T E M ---------------

OS: Windows Server 2003 family Build 3790 Service Pack 2

CPU:total 8 (4 cores per cpu, 1 threads per core) family 6 model 7 stepping 10, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

Memory: 4k page, physical 2097151k(2097151k free), swap 4194303k(4194303k free)

vm_info: Java HotSpot™ Server VM (11.0-b16) for windows-x86 JRE (1.6.0_11-b03), built on Nov 10 2008 02:10:33 by “java_re” with MS VC++ 7.1

time: Fri May 22 12:15:56 2009
elapsed time: 9943 seconds

Stefano Iacovella ha scritto:

Dear all

I'm having some trouble with geoserver 1.7.4.
I'm using it on three servers all with the same stack:

MS Windows server 2003 Enterprise edition 32 bit
Sun JDK 1.6.0_11-b03
Apache Tomcat 6.0.18
Oracle client 10.2.0.3

Without any correlation with some specific WMS or WFS request the jvm crashes and tomcat service restart.

I can found an hh_xxxx.pid file and a sqlnet.log in the main folder of Tomcat. I didn't find anything relevant on geoserver.log.

The sqlnet.log is filled with error messages like this

***********************************************************************
Fatal NI connect error 12560, connecting to:
(DESCRIPTION=(ADDRESS=(PROTOCOL=BEQ)(PROGRAM=oracle)(ARGV0=oracleORCL)(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(SID=ORCL)(CID=(PROGRAM=C:\opt\Tomcat6018\bin\tomcat6.exe)(HOST=SITR-MAP-03)(USER=SYSTEM))))

  VERSION INFORMATION:
    TNS for 32-bit Windows: Version 10.2.0.3.0 - Production
    Oracle Bequeath NT Protocol Adapter for 32-bit Windows: Version 10.2.0.3.0 - Production
    Windows NT TCP/IP NT Protocol Adapter for 32-bit Windows: Version 10.2.0.3.0 - Production
  Time: 22-MAY-2009 09:18:12
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12560
    TNS-12560: TNS:protocol adapter error
    ns secondary err code: 0
    nt main err code: 530
    TNS-00530: Protocol adapter error
    nt secondary err code: 126
    nt OS err code: 0

I'm pasting the content of one of the hh_xxxx.pid log file. Is there any way to understand what exactly is

Thank you for any hint on how to debug this

Stefano Iacovella

#
# An unexpected error has been detected by Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6d204aa6, pid=1528, tid=1816
#
# Java VM: Java HotSpot(TM) Server VM (11.0-b16 mixed mode windows-x86)
# Problematic frame:
# C [tcnative-1.dll+0x4aa6]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

--------------- T H R E A D ---------------

Current thread (0x6bee2400): JavaThread "Finalizer" daemon [_thread_in_native, id=1816, stack(0x6c370000,0x6c3c0000)]

siginfo: ExceptionCode=0xc0000005, reading address 0x00000020

Registers:
EAX=0x00000000, EBX=0x0000000e, ECX=0x00000000, EDX=0x00000000
ESP=0x6c3bf610, EBP=0x6c3bf628, ESI=0x00000000, EDI=0x71389788
EIP=0x6d204aa6, EFLAGS=0x00010246

Top of Stack: (sp=0x6c3bf610)
0x6c3bf610: 00000000 00000000 6c3bf630 6bee2400
0x6c3bf620: 042cdb00 042cdb00 6c3bf66c 00d3a27e
0x6c3bf630: 0000000e 6c3bf674 71389788 00000000
0x6c3bf640: 00000000 0000000e 05dfc430 6c3bf64c
0x6c3bf650: 00000000 6c3bf688 042ce7b8 00000000
0x6c3bf660: 042cdb00 00000000 6c3bf67c 6c3bf6ac
0x6c3bf670: 00d32f85 042ce758 00d38299 0000000e
0x6c3bf680: 00000000 71389788 00000000 6c3bf68c

Instructions: (pc=0x6d204aa6)
0x6d204a96: 55 18 51 8d 0c 16 8b 57 10 03 ca 8b 57 0c 51 52
0x6d204aa6: ff 50 20 85 c0 75 14 03 75 08 3b f3 72 d4 85 c0

Stack: [0x6c370000,0x6c3c0000], sp=0x6c3bf610, free space=317k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [tcnative-1.dll+0x4aa6]

It seems to be crashing in tomcat specific code, in particular
in the tcnative library.
I'd suggest your remove it (has happened to me in the
past as well, but on Linux... on some distributions it
works fine, on some others it just crashes under even
light load).

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

Thanks Andrea for your suggestion.
I checked all the hh_xxxx.pid in the tomcat folder. There are 22 files at the moment.
The header is of type

Java VM: Java HotSpot™ Server VM (11.0-b16 mixed mode windows-x86)

Problematic frame:

C [tcnative-1.dll+0x4aa6]

in four files

Java VM: Java HotSpot™ Server VM (11.0-b16 mixed mode windows-x86)

Problematic frame:

C [ocijdbc10.dll+0xdebf]

in one file

Java VM: Java HotSpot™ Server VM (11.0-b16 mixed mode windows-x86)

Problematic frame:

C [orageneric10.dll+0x4257e5]

in seven files

java.lang.OutOfMemoryError: requested 931272 bytes for Chunk::new. Out of swap space?

Internal Error (allocation.cpp:218), pid=1840, tid=3452

Error: Chunk::new

in seven files

Java VM: Java HotSpot™ Server VM (11.0-b16 mixed mode windows-x86)

Problematic frame:

V [jvm.dll+0x123d78]

in the remaining three files

I guess there is something wrong n the way geoserver connects to Oracle. Randomly I’m having connection errors like this

ORA-12571: TNS:packet writer failure

I’m using the standard oracle plugin to connect to Oracle.
On the same servers ArcIMS is installed and it also has problems connecting to Oracle

The strange thing is that on the same subnet I have three other server that are identical as hardware, OS, JDK, Tomcat release and Oracle client. On those server I deployed several web application connecting to Oracle and there are no errors in the connection.

Thank you

Stefano

The strange things

2009/5/22 Andrea Aime <aaime@anonymised.com>

Stefano Iacovella ha scritto:

Dear all

I’m having some trouble with geoserver 1.7.4.
I’m using it on three servers all with the same stack:

MS Windows server 2003 Enterprise edition 32 bit
Sun JDK 1.6.0_11-b03
Apache Tomcat 6.0.18
Oracle client 10.2.0.3

Without any correlation with some specific WMS or WFS request the jvm crashes and tomcat service restart.

I can found an hh_xxxx.pid file and a sqlnet.log in the main folder of Tomcat. I didn’t find anything relevant on geoserver.log.

The sqlnet.log is filled with error messages like this


Fatal NI connect error 12560, connecting to:
(DESCRIPTION=(ADDRESS=(PROTOCOL=BEQ)(PROGRAM=oracle)(ARGV0=oracleORCL)(ARGS=‘(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))’))(CONNECT_DATA=(SID=ORCL)(CID=(PROGRAM=C:\opt\Tomcat6018\bin\tomcat6.exe)(HOST=SITR-MAP-03)(USER=SYSTEM))))

VERSION INFORMATION:
TNS for 32-bit Windows: Version 10.2.0.3.0 - Production
Oracle Bequeath NT Protocol Adapter for 32-bit Windows: Version 10.2.0.3.0 - Production
Windows NT TCP/IP NT Protocol Adapter for 32-bit Windows: Version 10.2.0.3.0 - Production
Time: 22-MAY-2009 09:18:12
Tracing not turned on.
Tns error struct:
ns main err code: 12560
TNS-12560: TNS:protocol adapter error
ns secondary err code: 0
nt main err code: 530
TNS-00530: Protocol adapter error
nt secondary err code: 126
nt OS err code: 0

I’m pasting the content of one of the hh_xxxx.pid log file. Is there any way to understand what exactly is

Thank you for any hint on how to debug this

Stefano Iacovella

An unexpected error has been detected by Java Runtime Environment:

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6d204aa6, pid=1528, tid=1816

Java VM: Java HotSpot™ Server VM (11.0-b16 mixed mode windows-x86)

Problematic frame:

C [tcnative-1.dll+0x4aa6]

If you would like to submit a bug report, please visit:

http://java.sun.com/webapps/bugreport/crash.jsp

The crash happened outside the Java Virtual Machine in native code.

See problematic frame for where to report the bug.

--------------- T H R E A D ---------------

Current thread (0x6bee2400): JavaThread “Finalizer” daemon [_thread_in_native, id=1816, stack(0x6c370000,0x6c3c0000)]

siginfo: ExceptionCode=0xc0000005, reading address 0x00000020

Registers:
EAX=0x00000000, EBX=0x0000000e, ECX=0x00000000, EDX=0x00000000
ESP=0x6c3bf610, EBP=0x6c3bf628, ESI=0x00000000, EDI=0x71389788
EIP=0x6d204aa6, EFLAGS=0x00010246

Top of Stack: (sp=0x6c3bf610)
0x6c3bf610: 00000000 00000000 6c3bf630 6bee2400
0x6c3bf620: 042cdb00 042cdb00 6c3bf66c 00d3a27e
0x6c3bf630: 0000000e 6c3bf674 71389788 00000000
0x6c3bf640: 00000000 0000000e 05dfc430 6c3bf64c
0x6c3bf650: 00000000 6c3bf688 042ce7b8 00000000
0x6c3bf660: 042cdb00 00000000 6c3bf67c 6c3bf6ac
0x6c3bf670: 00d32f85 042ce758 00d38299 0000000e
0x6c3bf680: 00000000 71389788 00000000 6c3bf68c

Instructions: (pc=0x6d204aa6)
0x6d204a96: 55 18 51 8d 0c 16 8b 57 10 03 ca 8b 57 0c 51 52
0x6d204aa6: ff 50 20 85 c0 75 14 03 75 08 3b f3 72 d4 85 c0

Stack: [0x6c370000,0x6c3c0000], sp=0x6c3bf610, free space=317k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [tcnative-1.dll+0x4aa6]

It seems to be crashing in tomcat specific code, in particular
in the tcnative library.
I’d suggest your remove it (has happened to me in the
past as well, but on Linux… on some distributions it
works fine, on some others it just crashes under even
light load).

Cheers
Andrea


Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

Stefano Iacovella ha scritto:

Thanks Andrea for your suggestion.
I checked all the hh_xxxx.pid in the tomcat folder. There are 22 files at the moment.
The header is of type

# Java VM: Java HotSpot(TM) Server VM (11.0-b16 mixed mode windows-x86)
# Problematic frame:
# C [tcnative-1.dll+0x4aa6]

in four files

# Java VM: Java HotSpot(TM) Server VM (11.0-b16 mixed mode windows-x86)
# Problematic frame:
# C [ocijdbc10.dll+0xdebf]

in one file

# Java VM: Java HotSpot(TM) Server VM (11.0-b16 mixed mode windows-x86)
# Problematic frame:
# C [orageneric10.dll+0x4257e5]

in seven files

# java.lang.OutOfMemoryError: requested 931272 bytes for Chunk::new. Out of swap space?
#
# Internal Error (allocation.cpp:218), pid=1840, tid=3452
# Error: Chunk::new
#

in seven files

# Java VM: Java HotSpot(TM) Server VM (11.0-b16 mixed mode windows-x86)
# Problematic frame:
# V [jvm.dll+0x123d78]

in the remaining three files

I guess there is something wrong n the way geoserver connects to Oracle. Randomly I'm having connection errors like this

ORA-12571: TNS:packet writer failure

I'm using the standard oracle plugin to connect to Oracle.

Start removing native libraries up until you get a stable
enviroment. I suggest you try to use the Oracle thin driver
if you already removed the native tomcat extensions.

On the same servers ArcIMS is installed and it also has problems connecting to Oracle

The strange thing is that on the same subnet I have three other server that are identical as hardware, OS, JDK, Tomcat release and Oracle client. On those server I deployed several web application connecting to Oracle and there are no errors in the connection.

The kind of workload that a GIS application imposes on the
VM is completely different than a standard business application,
in general it's more stressful.
GeoServer and ArcIMS have completely different
code bases, but perform a similar kind of workload.
It may be the workload, or something else specific to that machine
(from a rootkit to a failing memory bank, I have no idea).
Hard to guess...

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

2009/5/22 Andrea Aime <aaime@anonymised.com>

Start removing native libraries up until you get a stable
enviroment. I suggest you try to use the Oracle thin driver
if you already removed the native tomcat extensions.

I see there is a tcnative-1.dll file in <tomcat_root>\bin folder. It is safe to remove it and restart the Tomcat or I need to update the configuration files?

On the same servers ArcIMS is installed and it also has problems connecting to Oracle

The strange thing is that on the same subnet I have three other server that are identical as hardware, OS, JDK, Tomcat release and Oracle client. On those server I deployed several web application connecting to Oracle and there are no errors in the connection.

The kind of workload that a GIS application imposes on the
VM is completely different than a standard business application,
in general it’s more stressful.
GeoServer and ArcIMS have completely different
code bases, but perform a similar kind of workload.
It may be the workload, or something else specific to that machine
(from a rootkit to a failing memory bank, I have no idea).
Hard to guess…

Yes really hard to guess. I don’t think it could be an hardware issue as it would be the same on three machines!

Thank you

Cheers
Andrea


Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

Stefano Iacovella ha scritto:

2009/5/22 Andrea Aime <aaime@anonymised.com <mailto:aaime@anonymised.com>>

    Start removing native libraries up until you get a stable
    enviroment. I suggest you try to use the Oracle thin driver
    if you already removed the native tomcat extensions.

I see there is a tcnative-1.dll file in <tomcat_root>\bin folder. It is safe to remove it and restart the Tomcat or I need to update the configuration files?

As far as I remember, it's safe, but it's been a while since I last
used the native extension.

        On the same servers ArcIMS is installed and it also has problems
        connecting to Oracle

        The strange thing is that on the same subnet I have three other
        server that are identical as hardware, OS, JDK, Tomcat release
        and Oracle client. On those server I deployed several web
        application connecting to Oracle and there are no errors in the
        connection.

    The kind of workload that a GIS application imposes on the
    VM is completely different than a standard business application,
    in general it's more stressful.
    GeoServer and ArcIMS have completely different
    code bases, but perform a similar kind of workload.
    It may be the workload, or something else specific to that machine
    (from a rootkit to a failing memory bank, I have no idea).
    Hard to guess...

Yes really hard to guess. I don't think it could be an hardware issue as it would be the same on three machines!

Oh, sorry, misread your mail, I thought it was happening on a single
machine only.
Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

2009/5/27 Andrea Aime <aaime@anonymised.com>

Stefano Iacovella ha scritto:

2009/5/22 Andrea Aime <aaime@anonymised.com mailto:[aaime@anonymised.com](mailto:aaime@anonymised.com)>

Start removing native libraries up until you get a stable
enviroment. I suggest you try to use the Oracle thin driver
if you already removed the native tomcat extensions.

I see there is a tcnative-1.dll file in <tomcat_root>\bin folder. It is safe to remove it and restart the Tomcat or I need to update the configuration files?

As far as I remember, it’s safe, but it’s been a while since I last
used the native extension.

OK I removed the dll and it seems the servers are ok. The number of hh_xxxx.pid files dropped significantly. There are yet a lot of oracle errors but I’m not sure geoserver plugin is involved.
Thank you again, if there will be any hint relevant to geoserver users I’ll post them on the mailing list.

Stefano