Dear all,
I am trying to deploy GeoNetwork 4.0.6 on Kubernetes with Elasticseach and Kibana support.
I've basically adopted the configuration on docker-compose.yml for GeoNetqork 4.06 on DockerHub and translated it into a K8S manifest file.
Here you are the most important pieces of K8S yaml file:
- - - - - - -
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: geonetwork
name: geonetwork-depl
labels:
app: geonetwork-depl
spec:
replicas: 1
selector:
matchLabels:
app: geonetwork-depl
template:
metadata:
labels:
app: geonetwork-depl
spec:
initContainers:
- name: check-db-ready
image: postgres:9.6.5
command: ['sh', '-c',
'until pg_isready -h postgres-svc -p 5432;
do echo waiting for database; sleep 2; done;']
containers:
- name: elasticsearch
image: elasticsearch:7.11.1
ports:
- name: elastic-srv
containerPort: 9200
protocol: TCP
- name: elastic-cluster
containerPort: 9300
protocol: TCP
env:
- name: ES_JAVA_OPTS
value: -Xms1G -Xmx1G
- name: discovery.type
value: single-node
volumeMounts:
- mountPath: /usr/share/elasticsearch/data
name: elasticsearch-data
- name: kibana
image: kibana:7.11.1
ports:
- name: kibana-srv
containerPort: 5601
protocol: TCP
env:
- name: SERVER_NAME
value: kibana
- name: SERVER_HOST
value: 0.0.0.0
- name: ELASTICSEARCH_HOSTS
value: http://127.0.0.1:9200
- name: SERVER_BASEPATH
value: /geonetwork/dashboards
- name: SERVER_REWRITEBASEPATH
value: 'false'
- name: KIBANA_INDEX
value: .dashboards
- name: XPACK_MONITORING_UI_CONTAINER_ELASTICSEARCH_ENABLED
value: 'true'
- name: geonetwork-core
image: geonetwork:4.0.6
ports:
- name: tomcat-http
containerPort: 8080
protocol: TCP
env:
- name: DATA_DIR
value: /catalogue-data
- name: JAVA_OPTS
value: >-
-Dorg.eclipse.jetty.annotations.AnnotationParser.LEVEL=OFF
-Djava.security.egd=file:/dev/./urandom -Djava.awt.headless=true
-Xms512M -Xss512M -Xmx2G -XX:+UseConcMarkSweepGC
-Dgeonetwork.resources.dir=/catalogue-data/resources
-Dgeonetwork.data.dir=/catalogue-data
-Dgeonetwork.codeList.dir=/var/lib/jetty/webapps/geonetwork/WEB-INF/data/config/codelist
-Dgeonetwork.schema.dir=/var/lib/jetty/webapps/geonetwork/WEB-INF/data/config/schema_plugins
- name: GEONETWORK_DB_HOST
value: postgres-svc
- name: GEONETWORK_DB_PORT
value: '5432'
- name: GEONETWORK_DB_NAME
value: geonetwork
- name: GEONETWORK_DB_USERNAME
value: geonetwork
- name: GEONETWORK_DB_PASSWORD
value: XXXXXXXXXXX
- name: GEONETWORK_DB_TYPE
value: postgres
- name: ES_HOST
value: 127.0.0.1
- name: ES_PROTOCOL
value: http
- name: ES_PORT
value: '9200'
- name: KB_URL
value: http://127.0.0.1:5601
volumeMounts:
- mountPath: /catalogue-data
name: geonetwork-data
- mountPath: /tmp/geonetwork-web-inf/config-security/config-security.xml
name: geonetwork-sec-conf
subPath: config-security.xml
- mountPath: /tmp/geonetwork-web-inf/config-security/config-security-ldap-recursive.xml
name: geonetwork-ldap-conf
subPath: config-security-ldap-recursive.xml
- mountPath: /tmp/geonetwork-web-inf/web.xml
name: geonetwork-webxml
subPath: web.xml
- mountPath: /tmp/geonetwork-web-inf/encryptor.properties
name: geonetwork-encryptor
subPath: encryptor.properties
- mountPath: /usr/local/tomcat/conf/
name: tomcat-conf
- mountPath: /usr/local/tomcat/conf/Catalina/localhost/
name: catalina-localhost
lifecycle:
postStart:
exec:
command:
- "/bin/sh"
- "-c"
- >
cp -f /tmp/geonetwork-web-inf/config-security/* /var/lib/jetty/webapps/geonetwork/WEB-INF/config-security/;
cp -f /tmp/geonetwork-web-inf/web.xml /var/lib/jetty/webapps/geonetwork/WEB-INF/web.xml;
cp -f /tmp/geonetwork-web-inf/encryptor.properties /var/lib/jetty/webapps/geonetwork/WEB-INF/data/config/encryptor.properties;
restartPolicy: Always
volumes:
...
- - - - - - - - -
The deployment seems to go fine and I can login into GeoNetwork GUI using my LDAP credentials, but when I click on "Content Statistics" in "Admin Console" -> "Statistics and status" I cannot see Kibana dashboard in the frame, and if I capture the request and Tomcat response I see a "frame-ancestors 'none" CSP policy that probably caused the DENY response, here's the full tcpdump capture:
- - - - - - - -
.,.~:4.CHTTP/1.1 500 Server Error
X-Frame-Options: DENY
Content-Security-Policy: frame-ancestors 'none'
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html;charset=iso-8859-1
Content-Length: 7698
Connection: close
Server: Jetty(9.4.45.v20220203)
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 java.lang.NullPointerException</title>
</head>
<body><h2>HTTP ERROR 500 java.lang.NullPointerException</h2>
<table>
<tr><th>URI:</th><td>/geonetwork/dashboards/s/geonetwork/app/kibana</td></tr>
<tr><th>STATUS:</th><td>500</td></tr>
<tr><th>MESSAGE:</th><td>java.lang.NullPointerException</td></tr>
<tr><th>SERVLET:</th><td>HttpDashboardProxy</td></tr>
<tr><th>CAUSED BY:</th><td>java.lang.NullPointerException</td></tr>
</table>
<h3>Caused by:</h3><pre>java.lang.NullPointerException
at org.mitre.dsmiley.httpproxy.ProxyServlet.copyRequestHeader(ProxyServlet.java:410)
at org.mitre.dsmiley.httpproxy.ProxyServlet.copyRequestHeaders(ProxyServlet.java:385)
at org.mitre.dsmiley.httpproxy.ProxyServlet.service(ProxyServlet.java:258)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1459)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631)
at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:230)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.fao.geonet.web.CORSResponseFilter.doFilter(CORSResponseFilter.java:129)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.fao.geonet.monitor.webapp.WebappMetricsFilter.doFilter(WebappMetricsFilter.java:121)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.fao.geonet.monitor.webapp.MetricsRegistryInitializerFilter.doFilter(MetricsRegistryInitializerFilter.java:58)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.fao.geonet.web.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:110)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176)
at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:394)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)
at org.fao.geonet.security.AuthenticathedUserFilter.doFilter(AuthenticathedUserFilter.java:17)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:155)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
at jeeves.config.springutil.JeevesDelegatingFilterProxy.doFilter(JeevesDelegatingFilterProxy.java:104)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(C
15:59:03.820375 IP 10.104.194.95.8080 > 10.104.126.45.51418: Flags [P.], seq 41412:42239, ack 11213, win 424, options [nop,nop,TS val 3173780606 ecr 976523843], length 827: HTTP
E..o..@anonymised.com@..s
h._
h~-........e`J.....X......
.,.~:4.ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.lang.Thread.run(Thread.java:750)
</pre>
<hr/><a href="https://eclipse.org/jetty">Powered by Jetty:// 9.4.45.v20220203</a><hr/>
</body>
</html>
- - - - - - - - - - -
Moreover, if I try to check Kibana status directly from http://localhost:5601, here's the (404 Not Found) response of the server:
- - - - - -
h._.n........e.....p......
g....t.sGET / HTTP/1.1
Host: localhost:5601
Connection: keep-alive
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="99", "Google Chrome";v="99"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.83 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,it;q=0.8
Cookie: ui_language=en_US
16:10:24.327421 IP 10.104.194.95.5601 > 10.0.150.101.52846: Flags [P.], seq 738:1074, ack 2134, win 303, options [nop,nop,TS val 2893412483 ecr 1740641433], length 336
E.....@anonymised.com@...
h._
..e...n..e......../n......
.u..g...HTTP/1.1 302 Found
location: /geonetwork/dashboards/spaces/enter
kbn-name: kibana
kbn-license-sig: 9697c476d1852e065a1ece447ec6b0750345fd95f7440a076084ede030cf68de
cache-control: private, no-cache, no-store, must-revalidate
content-length: 0
Date: Tue, 29 Mar 2022 14:10:24 GMT
Connection: keep-alive
Keep-Alive: timeout=120
16:10:24.327528 IP 10.0.150.101.52846 > 10.104.194.95.5601: Flags [.], ack 1074, win 284, options [nop,nop,TS val 1740641445 ecr 2893412483], length 0
E..4y.@anonymised.com@.T.
..e
h._.n........f.....mS.....
g....u..
16:10:24.443800 IP 10.0.150.101.52846 > 10.104.194.95.5601: Flags [P.], seq 2134:2868, ack 1074, win 284, options [nop,nop,TS val 1740641561 ecr 2893412483], length 734
E...y.@anonymised.com@.Q.
..e
h._.n........f.....p1.....
g....u..GET /geonetwork/dashboards/spaces/enter HTTP/1.1
Host: localhost:5601
Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.83 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="99", "Google Chrome";v="99"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,it;q=0.8
Cookie: ui_language=en_US
- - - - -
The Elasticsearch service is up and the indexes status is:
- - - - - - -
$ curl -XGET http://localhost:9200/_cat/indices
green open .apm-custom-link 5Hq-XfZaQgi5IuaomGkyFA 1 0 0 0 208b 208b
green open .kibana_task_manager_1 50YtCkgVTWy7Ka72vKkBJg 1 0 8 849 187.4kb 187.4kb
yellow open gn-features WtDbkOA_Q3SaGkMsBJ0TpA 1 1 0 0 208b 208b
green open .apm-agent-configuration AGPARm6zSViImCE9wAu3TA 1 0 0 0 208b 208b
green open .dashboards_1 VU-xPuDEReKpL0lYX49h6g 1 0 11 0 2.1mb 2.1mb
yellow open gn-records V0wTMf2oQ2qsvLywVMzC0g 1 1 147 0 820.7kb 820.7kb
green open .dashboards-event-log-7.11.1-000001 8iyArYvFS-mviXNVlwTSFQ 1 0 0 0 208b 208b
yellow open gn-searchlogs aIuwpjxTRlSDMCYnj2OxjA 1 1 0 0 208b 208b
- - - - - -
Can someone help me to identify the problem in my deployment?
Best regards,
Pierpaolo
(attachments)
