[Geoserver-devel] Sigh - not able to run Postgis test

Okay I am with brent and we tried running our overlap test:

We have attached:
data/plugIns/OverlapsIntegrity.xml
data/validation/test.xml

These illustrate the use of the OverlapsIntegrity.java - you will need to grab an update from geotools trunk (brents code was not passing its junit tests).

So the result of this is that our testing is - we could not run our test. Something else seems to be broken? If you look at the following strack trace our code does not seem to be mentioned at all?

2004-06-17 15:35:49 ApplicationDispatcher[/geoserver] Servlet.service() for servlet jsp threw exception
javax.servlet.jsp.JspException: Property namespaces returned a null value
    at org.apache.struts.taglib.html.OptionsTag.getIterator(OptionsTag.java:416)
    at org.apache.struts.taglib.html.OptionsTag.doEndTag(OptionsTag.java:289)
    at org.apache.jsp.WEB_002dINF.pages.data.namespaces.Select_jsp._jspx_meth_html_options_0(Select_jsp.java:215)
    at org.apache.jsp.WEB_002dINF.pages.data.namespaces.Select_jsp._jspx_meth_html_select_0(Select_jsp.java:189)
    at org.apache.jsp.WEB_002dINF.pages.data.namespaces.Select_jsp._jspx_meth_html_form_0(Select_jsp.java:113)
    at org.apache.jsp.WEB_002dINF.pages.data.namespaces.Select_jsp._jspService(Select_jsp.java:75)
    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:133)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:311)
    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:301)
    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:248)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:284)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:204)
    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:750)
    at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:636)
    at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:546)
    at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:1002)
    at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:626)
    at org.apache.struts.tiles.TilesUtilImpl.doInclude(TilesUtilImpl.java:137)
    at org.apache.struts.tiles.TilesUtil.doInclude(TilesUtil.java:177)
    at org.apache.struts.taglib.tiles.InsertTag.doInclude(InsertTag.java:756)
    at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:881)
    at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:473)
    at org.apache.jsp.WEB_002dINF.pages.layouts.mainLayout_jsp._jspx_meth_tiles_insert_6(mainLayout_jsp.java:1337)
    at org.apache.jsp.WEB_002dINF.pages.layouts.mainLayout_jsp._jspService(mainLayout_jsp.java:519)
    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:133)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:311)
    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:301)
    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:248)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:284)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:204)
    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:750)
    at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:510)
    at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:445)
    at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:359)
    at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1069)
    at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:274)
    at org.apache.struts.tiles.TilesRequestProcessor.processTilesDefinition(TilesRequestProcessor.java:254)
    at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(TilesRequestProcessor.java:309)
    at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:279)
    at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
    at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:743)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:284)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:204)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:257)
    at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:151)
    at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:567)
    at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:245)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:199)
    at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:151)
    at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:567)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:184)
    at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:151)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:164)
    at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:149)
    at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:567)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:156)
    at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:151)
    at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:567)
    at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:972)
    at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:206)
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:833)
    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:732)
    at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:619)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:688)
    at java.lang.Thread.run(Thread.java:534)

2004-06-17 15:39:05 StandardWrapperValve[action]: Servlet.service() for servlet action threw exception
java.lang.OutOfMemoryError

(attachments)

test.xml (664 Bytes)
OverlapsIntegrity.xml (464 Bytes)

Same idea as before:

*description* _The server encountered an internal error () that prevented it from fulfilling this request._

*exception*

javax.servlet.ServletException: Servlet execution threw an exception
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2422)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:163)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:199)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:828)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:700)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:584)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
at java.lang.Thread.run(Thread.java:534)

You can use this as a guide when setting up a test against the postgis datastore.

Finally found a real error:

java.lang.NullPointerException
        at org.geotools.data.shapefile.ShapefileDataStore$Reader.close(ShapefileDataStore.java:399)
        at org.geotools.data.FIDFeatureReader.close(FIDFeatureReader.java:125)
        at org.geotools.data.FilteringFeatureReader.close(FilteringFeatureReader.java:82)
        at org.geotools.validation.relate.OverlapsIntegrity.validateSingleLayer(OverlapsIntegrity.java:368)
        at org.geotools.validation.relate.OverlapsIntegrity.validate(OverlapsIntegrity.java:118)
        at org.geotools.validation.ValidationProcessor.runIntegrityTests(ValidationProcessor.java:427)
        at org.vfny.geoserver.action.validation.ValidationTestDoIt.runTransactions(ValidationTestDoIt.java:108)

Ok, I've been working away for about four hours, but I'm now going to
have to shift my focus to other bugs, and probably leave this one for
after rc2, as I don't know the code that well, where the problems are and
whatnot. I played with JProbe for a bit, but couldn't really figure it
out to do what I wanted to (see what classes were causing the out of
memory errors).

But I've noticed really odd behavior with the test suites. And I've just
now confirmed that it happens with rc1 as well, so this is _not_ a problem
introduced with fid-exp or mm. Basically what looks to be happening is
that readers are being called on every single table in the postgis
datastore, _regardless_ of if they are defined in the test, the test
suite, or even in geoserver. Like for tables that aren't even official
featureTypes. Even if a datastore has no featureTypes at all defined the
validation processor is somewhere issueing calls to a reader, as sql
statements are being issued in the logs.

And it looks like the features aren't being thrown away right away, since
this is likely the cause of the Out of Memory errors. If I only add a
database that has a few small tables then it just iterates through them
(doing what I have no idea), and then works fine, no out of memory errors.
This occurs with Null Zero, so it's not just geometry, or relate type
tests. I noticed this behavior when I first started testing, as only one
sql call was getting issued for me (I have some big tables sitting
around). The relates should be issueing a bunch of calls.

I have no idea about the validation code, why it might be trying to hit
all the featureTypes in a DataStore. I can't think of a good reason for
it to though.

So once again I'm not actually at the problem of issueing too many postgis
calls or getting too many result sets.

...

Ok, I successfully ran an overlaps test against postgis. It took maybe 20
minutes, and issued a ton of calls with that lovely loop, but it seemed to
work, with no out of memory errors. I ran it against buildings.shp
converted to postgis (in a db with just buildings, and two other very,
very small tables, which seems necessary to avoid the out of memory
errors). So it looks like the new driver helps.

I personally think this validation test is pretty ridiculous though, as
postgis is just too slow. You should consider rewriting them to cache one
or both of the featureResults into a pickle datastore, as I think that'd
be a lot more efficient then making fifty million connections to postgis.

Chris

On Thu, 17 Jun 2004, Jody Garnett wrote:

You can use this as a guide when setting up a test against the postgis
datastore.

-------------------------------------------------------
This SF.Net email is sponsored by The 2004 JavaOne(SM) Conference
Learn from the experts at JavaOne(SM), Sun's Worldwide Java Developer
Conference, June 28 - July 1 at the Moscone Center in San Francisco, CA
REGISTER AND SAVE! http://java.sun.com/javaone/sf Priority Code NWMGYKND
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--