Hi List,
Apologies for a really long email, I have two big problems and need some
pointers on where to go for more information about the errors, or some
ideas of how to fix.
I recently upgraded a Geonetwork 2.10.4 installation to the latest snapshot
to apply some security patches. I'm using tomcat6 and my approach was to
create a new database (postgresql) and restore a snapshot of my live one to
that. I copied my live data directory to a new one. I deployed a renamed
war file and set the config to point at the new database and data
directory. This all worked fine, so I stopped tomcat6, and renamed
everything so that the new version was the one in use. The configuration
all seemed fine for a couple of days but then I started to get lots of
problems with both the spatial index and the lucene index.
Firstly the spatial index- previously I was using postgresql for my
database but a shapefile spatial index. Fairly quickly after doing the
upgrade I started to see errors about corruptions in the index.
The metadata document index with id=5917 is corrupt/invalid - ignoring it.
Error: null
2015-07-20 02:06:35,530 ERROR [geonetwork.index] - Failed to properly index
geometry of metadata 5918. Error: Dbf has extra record
java.io.IOException: Dbf has extra record
at
org.geotools.data.shapefile.ShapefileAttributeReader.internalReadersHaveNext(ShapefileAttributeReader.java:153)
at
org.geotools.data.shapefile.ShapefileFeatureWriter.close(ShapefileFeatureWriter.java:237)
at
org.geotools.data.shapefile.indexed.IndexedShapefileFeatureWriter.close(IndexedShapefileFeatureWriter.java:109)
at
org.geotools.data.InProcessLockingManager$1.close(InProcessLockingManager.java:350)
at
org.geotools.data.FilteringFeatureWriter.close(FilteringFeatureWriter.java:151)
at
org.geotools.data.AbstractFeatureStore.removeFeatures(AbstractFeatureStore.java:370)
at
org.fao.geonet.kernel.search.spatial.SpatialIndexWriter.delete(SpatialIndexWriter.java:221)
at
org.fao.geonet.kernel.search.SearchManager.indexGeometry(SearchManager.java:711)
at org.fao.geonet.kernel.search.SearchManager.index(SearchManager.java:695)
at org.fao.geonet.kernel.DataManager.indexMetadata(DataManager.java:580)
at
org.fao.geonet.kernel.DataManager$IndexMetadataTask.run(DataManager.java:399)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
As I couldn't see any documentation on how I get around this, I decided to
convert to a postgis index so I altered the connection string to
jdbc:postgresql_postGIS://host:5432/database. The documentation suggests
this should be enough to get Geonetwork to use postgis for the spatial
index. I created the spatialindex table using the setup scripts and when I
restarted Geonetwork this table was populated. However, when my server has
restarted since then, it has reverted to using shapefiles for it's index
and I'm seeing the corruption again. I can't see any errors about why it
would revert back to using shapefiles.
Secondly, I'm also seeing corruption in the lucene index. These are the log
entries at startup:
2015-07-20 02:05:46,719 INFO [jeeves.apphand] - Lucene configuration:
* Version: LUCENE_36
* RAMBufferSize: 48.0
* MergeFactor: 10
* Default analyzer: org.apache.lucene.analysis.WhitespaceAnalyzer
* Field analyzers:
{_source=org.apache.lucene.analysis.core.KeywordAnalyzer,
hassource=org.fao.geonet.kernel.search.GeoNetworkAnalyzer,
parentUuid=org.fao.geonet.kernel.search.GeoNetworkAnalyzer,
_groupPublished=org.apache.lucene.analysis.core.KeywordAnalyzer,
subject=org.apache.lucene.analysis.core.KeywordAnalyzer,
hasfeaturecat=org.fao.geonet.kernel.search.GeoNetworkAnalyzer,
keyword=org.apache.lucene.analysis.core.KeywordAnalyzer,
any=org.fao.geonet.kernel.search.GeoNetworkAnalyzer,
_cat=org.apache.lucene.analysis.core.KeywordAnalyzer,
operatesOnIdentifier=org.fao.geonet.kernel.search.GeoNetworkAnalyzer,
operatesOn=org.fao.geonet.kernel.search.GeoNetworkAnalyzer,
_uuid=org.fao.geonet.kernel.search.GeoNetworkAnalyzer,
inspirerelated=org.apache.lucene.analysis.core.KeywordAnalyzer}
* Field search analyzers:
{_source=org.apache.lucene.analysis.core.KeywordAnalyzer,
parentUuid=org.fao.geonet.kernel.search.GeoNetworkAnalyzer,
hassource=org.fao.geonet.kernel.search.GeoNetworkAnalyzer,
subject=org.apache.lucene.analysis.core.KeywordAnalyzer,
_groupPublished=org.apache.lucene.analysis.core.KeywordAnalyzer,
keyword=org.apache.lucene.analysis.core.KeywordAnalyzer,
operatesOn=org.fao.geonet.kernel.search.GeoNetworkAnalyzer,
hasfeaturecat=org.fao.geonet.kernel.search.GeoNetworkAnalyzer,
any=org.apache.lucene.analysis.WhitespaceAnalyzer,
_cat=org.apache.lucene.analysis.core.KeywordAnalyzer,
operatesOnIdentifier=org.fao.geonet.kernel.search.GeoNetworkAnalyzer,
_uuid=org.fao.geonet.kernel.search.GeoNetworkAnalyzer,
inspirerelated=org.apache.lucene.analysis.core.KeywordAnalyzer}
* Field boost factor: {_op2=0.0, _owner=0.0, _isTemplate=0.0, _op0=0.0,
_op1=0.0, _dummy=0.0}
* Boost document class: null
* Tokenized fields: [abstract, keywordType, title, subject, any,
levelName, altTitle, specificationTitle]
* Numeric fields: [eastBL, southBL, northBL, westBL, denominator]
* Dump fields:
{_valid_schematron-rules-inspire=valid_schematron-rules-inspire,
_popularity=popularity, _dynamic=dynamic, _source=source, _owner=owner,
accessConstr=Constraints, link=link, _ownername=ownername, _view=view,
responsibleParty=responsibleParty, type=type, _userinfo=userinfo,
_locale=_locale, _valid_schematron-rules-iso=valid_schematron-rules-iso,
_selected=selected, _isTemplate=isTemplate,
_isPublishedToAll=isPublishedToAll, _valid=valid,
classif=SecurityConstraints, _rating=rating, _cat=category, credit=credit,
otherConstr=Constraints, _status=status, _title=title, abstract=abstract,
_featured=featured,
_valid_schematron-rules-geonetwork=valid_schematron-rules-geonetwork,
parentUuid=parentId, keyword=keyword, image=image, datasetLang=datasetLang,
conditionApplyingToAccessAndUse=Constraints, _displayOrder=displayOrder,
_indexingErrorMsg=idxMsg, serviceType=serviceType, geoBox=geoBox,
_isHarvested=isHarvested,
spatialRepresentationType=spatialRepresentationType, _download=download,
_valid_xsd=valid_xsd, _indexingError=idxError, language=language,
_defaultTitle=defaultTitle, _edit=edit, _docLocale=docLocale,
_notify=notify}
* Search boost query: null
* Score:
* trackDocScores: false
* trackMaxScore: false
* docsScoredInOrder: false
Taxonomy configuration: [hits, results, titles, results_with_summary]
* type: hits
{inspiretheme=Field: inspiretheme name:inspireTheme max:35 sort byVALUE sort
order:ASCENDIND, metadataPOC=Field: metadataPOC name:metadataPOC max:15 sort
byCOUNT sort order:DESCENDING, spatialRepresentationType=Field:
spatialRepresentationType name:spatialRepresentationType max:10 sort
byCOUNT sort
order:DESCENDING, keyword=Field: keyword name:keyword max:15 sort byCOUNT sort
order:DESCENDING, createDateYear=Field: createDateYear name:createDateYear
max:40 sort byVALUE sort order:DESCENDING, denominator=Field: denominator
name:denominator max:10 sort byNUMVALUE sort order:DESCENDING, type=Field:
type name:type max:10 sort byCOUNT sort order:DESCENDING,
serviceType=Field: serviceType name:serviceType max:10 sort byCOUNT sort
order:DESCENDING, orgName=Field: orgName name:orgName max:15 sort byCOUNT sort
order:DESCENDING} * type: results
{keyword=Field: keyword name:keyword max:100 sort byCOUNT sort
order:DESCENDING} * type: titles
{title=Field: title name:title max:100 sort byCOUNT sort order:DESCENDING}
* type: results_with_summary
{inspiretheme=Field: inspiretheme name:inspireTheme max:35 sort byVALUE sort
order:ASCENDIND, metadataPOC=Field: metadataPOC name:metadataPOC max:15 sort
byCOUNT sort order:DESCENDING, spatialRepresentationType=Field:
spatialRepresentationType name:spatialRepresentationType max:10 sort
byCOUNT sort
order:DESCENDING, keyword=Field: keyword name:keyword max:15 sort byCOUNT sort
order:DESCENDING, createDateYear=Field: createDateYear name:createDateYear
max:40 sort byCOUNT sort order:ASCENDIND, denominator=Field: denominator
name:denominator max:20 sort byVALUE sort order:ASCENDIND, type=Field: type
name:type max:10 sort byCOUNT sort order:DESCENDING, serviceType=Field:
serviceType name:serviceType max:10 sort byCOUNT sort order:DESCENDING,
orgName=Field: orgName name:orgName max:15 sort byCOUNT sort
order:DESCENDING}
2015-07-20 02:05:46,831 WARN [geonetwork.search] - Failed to create
analyzer as specified in lucene config, default analyzer will be used for
field null. Exception message is:
org.apache.lucene.analysis.WhitespaceAnalyzer
2015-07-20 02:05:46,832 WARN [geonetwork.search] - Creating analyzer has
failed, defaulting to GeoNetworkAnalyzer
2015-07-20 02:05:46,876 WARN [geonetwork.search] - Failed to create
analyzer as specified in lucene config, default analyzer will be used for
field null. Exception message is:
org.apache.lucene.analysis.WhitespaceAnalyzer
2015-07-20 02:05:46,877 WARN [geonetwork.search] - Creating analyzer has
failed, defaulting to GeoNetworkAnalyzer
(this continues for ~200 lines)
When people do a lucene search they get an error in the browser and I see
these sorts of errors in the log file:
2015-07-20 02:05:49,479 ERROR [geonetwork.index] - An error occurred while
openning taxonomy readers/writers
org.apache.lucene.index.CorruptIndexException: failed to locate current
segments_N file
at
org.apache.lucene.index.IndexFileDeleter.<init>(IndexFileDeleter.java:224)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:704)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.openIndexWriter(DirectoryTaxonomyWriter.java:283)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.<init>(DirectoryTaxonomyWriter.java:210)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.<init>(DirectoryTaxonomyWriter.java:332)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.<init>(DirectoryTaxonomyWriter.java:348)
at
org.fao.geonet.kernel.search.index.TaxonomyIndexTracker.init(TaxonomyIndexTracker.java:57)
at
org.fao.geonet.kernel.search.index.TaxonomyIndexTracker.<init>(TaxonomyIndexTracker.java:44)
at
org.fao.geonet.kernel.search.index.LuceneIndexLanguageTracker.<init>(LuceneIndexLanguageTracker.java:55)
at
org.fao.geonet.kernel.search.SearchManager.setupIndex(SearchManager.java:1318)
at
org.fao.geonet.kernel.search.SearchManager.initLucene(SearchManager.java:648)
at org.fao.geonet.kernel.search.SearchManager.<init>(SearchManager.java:542)
at org.fao.geonet.Geonetwork.start(Geonetwork.java:320)
at jeeves.server.JeevesEngine.initAppHandler(JeevesEngine.java:605)
at jeeves.server.JeevesEngine.init(JeevesEngine.java:206)
at jeeves.server.sources.http.JeevesServlet.init(JeevesServlet.java:91)
at javax.servlet.GenericServlet.init(GenericServlet.java:212)
at
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1213)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1026)
at
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4425)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4738)
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
at
org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
at
org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
at org.apache.catalina.core.StandardService.start(StandardService.java:525)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
2015-07-20 02:05:49,481 ERROR [geonetwork.lucene] - Failure while closing
luceneIndexLanguageTracker
java.lang.NullPointerException
at
org.fao.geonet.kernel.search.index.TaxonomyIndexTracker.close(TaxonomyIndexTracker.java:129)
at
org.fao.geonet.kernel.search.index.TaxonomyIndexTracker.init(TaxonomyIndexTracker.java:70)
at
org.fao.geonet.kernel.search.index.TaxonomyIndexTracker.<init>(TaxonomyIndexTracker.java:44)
at
org.fao.geonet.kernel.search.index.LuceneIndexLanguageTracker.<init>(LuceneIndexLanguageTracker.java:55)
at
org.fao.geonet.kernel.search.SearchManager.setupIndex(SearchManager.java:1318)
at
org.fao.geonet.kernel.search.SearchManager.initLucene(SearchManager.java:648)
at org.fao.geonet.kernel.search.SearchManager.<init>(SearchManager.java:542)
at org.fao.geonet.Geonetwork.start(Geonetwork.java:320)
at jeeves.server.JeevesEngine.initAppHandler(JeevesEngine.java:605)
at jeeves.server.JeevesEngine.init(JeevesEngine.java:206)
at jeeves.server.sources.http.JeevesServlet.init(JeevesServlet.java:91)
at javax.servlet.GenericServlet.init(GenericServlet.java:212)
at
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1213)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1026)
at
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4425)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4738)
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
at
org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
at
org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
at org.apache.catalina.core.StandardService.start(StandardService.java:525)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
2015-07-20 02:05:49,481 ERROR [geonetwork.index] - Exception while opening
lucene index, going to rebuild it:
org.apache.lucene.index.CorruptIndexException: failed to locate current
segments_N file
at
org.apache.lucene.index.IndexFileDeleter.<init>(IndexFileDeleter.java:224)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:704)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.openIndexWriter(DirectoryTaxonomyWriter.java:283)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.<init>(DirectoryTaxonomyWriter.java:210)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.<init>(DirectoryTaxonomyWriter.java:332)
at
org.apache.lucene.facet.taxonomy.directory.DirectoryTaxonomyWriter.<init>(DirectoryTaxonomyWriter.java:348)
at
org.fao.geonet.kernel.search.index.TaxonomyIndexTracker.init(TaxonomyIndexTracker.java:57)
at
org.fao.geonet.kernel.search.index.TaxonomyIndexTracker.<init>(TaxonomyIndexTracker.java:44)
at
org.fao.geonet.kernel.search.index.LuceneIndexLanguageTracker.<init>(LuceneIndexLanguageTracker.java:55)
at
org.fao.geonet.kernel.search.SearchManager.setupIndex(SearchManager.java:1318)
at
org.fao.geonet.kernel.search.SearchManager.initLucene(SearchManager.java:648)
at org.fao.geonet.kernel.search.SearchManager.<init>(SearchManager.java:542)
at org.fao.geonet.Geonetwork.start(Geonetwork.java:320)
at jeeves.server.JeevesEngine.initAppHandler(JeevesEngine.java:605)
at jeeves.server.JeevesEngine.init(JeevesEngine.java:206)
at jeeves.server.sources.http.JeevesServlet.init(JeevesServlet.java:91)
at javax.servlet.GenericServlet.init(GenericServlet.java:212)
at
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1213)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1026)
at
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4425)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4738)
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
at
org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
at
org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
at org.apache.catalina.core.StandardService.start(StandardService.java:525)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
2015-07-20 02:05:49,482 ERROR [geonetwork.index] - Rebuilding lucene index
Rebuilding the index from the admin menu fixes the problem, but it returns-
first it happened about 4 days later but most recently it happened on
Friday and then again this morning.
Can anyone help me with either of these problems?
Regards
Jo
--
*Jo Cook*
Astun Technology Ltd, The Coach House, 17 West Street, Epsom, Surrey, KT18
7RL, UK
t:+44 7930 524 155
iShare - Data integration and publishing platform
<http://www.isharemaps.com/>
*****************************************
Company registration no. 5410695. Registered in England and Wales.
Registered office: 120 Manor Green Road, Epsom, Surrey, KT19 8LN VAT no.
864201149.