[Geoserver-devel] GeoPackage Raster Mosaic tile order

I’ve discovered some interesting behaviour in the GeoPackage Raster Mosaic implementation over the last few days.

I tried viewing a GeoPackage generated from the GeoServer GeoPackage WMS output format in QGIS, and observed that tile order is reversed along the y-axis.

QGIS used GDAL for parsing GeoPackages. After running this issue by the GDAL team, I was directed to this line of the OGC GeoPackage spec:

The tile coordinate (0,0) always refers to the tile in the upper left corner of the tile matrix at any zoom level, regardless of the actual availability of that tile

Adding some test cases to the GeoTools GeoPackage plugin, I can confirm that GeoTools gets this order wrong. Among other things, this means that GeoTools / GeoServer will incorrectly read valid GeoPackage files. A fix can be found here: https://github.com/geotools/geotools/pull/1567

Moving along to GeoServer, it appears there are similar problems with the GeoServer GeoPackage Community module, namely in the GeoPackageGetMapOutputFormat.

I believe I have a fix, but am currently being slowed down by the fact that the GeoPackage module tests don’t seem to be passing on master.

This issue does bring up a problem: Because GeoTools / GeoServer is consistently wrong (a GeoPackage generated by GeoServer will be read in as “correct”), this bug fix could cause problems for anyone who has solely been using GeoPackage Rasters that were generated by GeoTools / GeoServer.

For more details:
https://osgeo-org.atlassian.net/browse/GEOT-5715
https://osgeo-org.atlassian.net/browse/GEOS-8112
https://trac.osgeo.org/gdal/ticket/6871

Torben

GeoServer side of the fix (WMS GeoPackage output format) here: https://github.com/geoserver/geoserver/pull/2285

I still get a couple of GeoPackage tests failing, however they are unrelated to the change and are also failing on master (and 2.11.x, 2.10.x, 2.9.x … perhaps this is only failing on my machine?). I’m not having much luck tracking down the root cause of those failures:

Tests in error:

testGetFeatureWithFilter(org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest): Error occurred creating table

testGetFeatureTwoTypes(org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest): Error occurred creating table

If anyone wants to test master locally, and see if those tests run for them or are failing, that would be appreciated.

Torben

···

On Wed, Apr 26, 2017 at 1:17 PM, Torben Barsballe <tbarsballe@anonymised.com> wrote:

I’ve discovered some interesting behaviour in the GeoPackage Raster Mosaic implementation over the last few days.

I tried viewing a GeoPackage generated from the GeoServer GeoPackage WMS output format in QGIS, and observed that tile order is reversed along the y-axis.

QGIS used GDAL for parsing GeoPackages. After running this issue by the GDAL team, I was directed to this line of the OGC GeoPackage spec:

The tile coordinate (0,0) always refers to the tile in the upper left corner of the tile matrix at any zoom level, regardless of the actual availability of that tile

Adding some test cases to the GeoTools GeoPackage plugin, I can confirm that GeoTools gets this order wrong. Among other things, this means that GeoTools / GeoServer will incorrectly read valid GeoPackage files. A fix can be found here: https://github.com/geotools/geotools/pull/1567

Moving along to GeoServer, it appears there are similar problems with the GeoServer GeoPackage Community module, namely in the GeoPackageGetMapOutputFormat.

I believe I have a fix, but am currently being slowed down by the fact that the GeoPackage module tests don’t seem to be passing on master.

This issue does bring up a problem: Because GeoTools / GeoServer is consistently wrong (a GeoPackage generated by GeoServer will be read in as “correct”), this bug fix could cause problems for anyone who has solely been using GeoPackage Rasters that were generated by GeoTools / GeoServer.

For more details:
https://osgeo-org.atlassian.net/browse/GEOT-5715
https://osgeo-org.atlassian.net/browse/GEOS-8112
https://trac.osgeo.org/gdal/ticket/6871

Torben

Torben,

I also see these failures locally on master:

Failed tests:
testTileEntries(org.geoserver.geopkg.GeoPackageGetMapOutputFormatTest)
Tests in error:
testGetFeatureWithFilter(org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest): Error occurred creating table
testGetFeatureTwoTypes(org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest): Error occurred creating table

The root cause is "java.sql.SQLException: The prepared statement has been finalized".

The first failure has the following stack trace :

Running org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest
Tests run: 5, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 4.402 sec <<< FAILURE!
testGetFeatureWithFilter(org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest) Time elapsed: 3505 sec <<< ERROR!
java.io.IOException: Error occurred creating table
  at org.geotools.jdbc.JDBCDataStore.createSchema(JDBCDataStore.java:800)
  at org.geotools.geopkg.GeoPackage.create(GeoPackage.java:629)
  at org.geotools.geopkg.GeoPackage.add(GeoPackage.java:654)
  at org.geoserver.geopkg.GeoPackageGetFeatureOutputFormat.write(GeoPackageGetFeatureOutputFormat.java:110)
  at org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest.testGetFeature(GeoPackageGetFeatureOutputFormatTest.java:137)
  at org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest.testGetFeatureWithFilter(GeoPackageGetFeatureOutputFormatTest.java:104)
[... omitting surefile/junit-related frames ...]
Caused by: java.sql.SQLException: java.io.IOException: java.sql.SQLException: The prepared statement has been finalized
  at org.geotools.geopkg.GeoPkgDialect.postCreateTable(GeoPkgDialect.java:302)
  at org.geotools.jdbc.JDBCDataStore.createSchema(JDBCDataStore.java:797)
  ... 39 more
Caused by: java.io.IOException: java.sql.SQLException: The prepared statement has been finalized
  at org.geotools.geopkg.GeoPackage.addGeometryColumnsEntry(GeoPackage.java:932)
  at org.geotools.geopkg.GeoPkgDialect.postCreateTable(GeoPkgDialect.java:297)
  ... 40 more
Caused by: java.sql.SQLException: The prepared statement has been finalized
  at org.sqlite.core.NativeDB.throwex(NativeDB.java:429)
  at org.sqlite.core.NativeDB.reset(Native Method)
  at org.sqlite.jdbc3.JDBC3PreparedStatement.execute(JDBC3PreparedStatement.java:57)
  at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
  at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
  at org.geotools.geopkg.GeoPackage.addGeometryColumnsEntry(GeoPackage.java:924)
  ... 41 more

Kind regards,
Ben.

On 27/04/17 11:29, Torben Barsballe wrote:

GeoServer side of the fix (WMS GeoPackage output format) here:
https://github.com/geoserver/geoserver/pull/2285

I still get a couple of GeoPackage tests failing, however they are
unrelated to the change and are also failing on master (and 2.11.x, 2.10.x,
2.9.x ... perhaps this is only failing on my machine?). I'm not having much
luck tracking down the root cause of those failures:

Tests in error:

  testGetFeatureWithFilter(org.geoserver.geopkg.
GeoPackageGetFeatureOutputFormatTest): Error occurred creating table

  testGetFeatureTwoTypes(org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest):
Error occurred creating table

If anyone wants to test master locally, and see if those tests run for them
or are failing, that would be appreciated.

Torben

On Wed, Apr 26, 2017 at 1:17 PM, Torben Barsballe <
tbarsballe@anonymised.com> wrote:

I've discovered some interesting behaviour in the GeoPackage Raster Mosaic
implementation over the last few days.

I tried viewing a GeoPackage generated from the GeoServer GeoPackage WMS
output format in QGIS, and observed that tile order is reversed along the
y-axis.

QGIS used GDAL for parsing GeoPackages. After running this issue by the
GDAL team, I was directed to this line of the OGC GeoPackage spec
<http://www.geopackage.org/spec/#tile_matrix&gt;:

The tile coordinate (0,0) always refers to the tile in the upper left

corner of the tile matrix at any zoom level, regardless of the actual
availability of that tile

Adding some test cases to the GeoTools GeoPackage plugin, I can confirm
that GeoTools gets this order wrong. Among other things, this means that
GeoTools / GeoServer will incorrectly read valid GeoPackage files. A fix
can be found here: https://github.com/geotools/geotools/pull/1567

Moving along to GeoServer, it appears there are similar problems with the
GeoServer GeoPackage Community module, namely in
the GeoPackageGetMapOutputFormat.

I believe I have a fix, but am currently being slowed down by the fact
that the GeoPackage module tests don't seem to be passing on master.

*This issue does bring up a problem*: Because GeoTools / GeoServer is
consistently wrong (a GeoPackage generated by GeoServer will be read in as
"correct"), this bug fix could cause problems for anyone who has solely
been using GeoPackage Rasters that were generated by GeoTools / GeoServer.

For more details:
https://osgeo-org.atlassian.net/browse/GEOT-5715
https://osgeo-org.atlassian.net/browse/GEOS-8112
https://trac.osgeo.org/gdal/ticket/6871

Torben

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

Torben,

the "java.sql.SQLException: The prepared statement has been finalized" is caused by failed error handling trying to double-finalize the prepared statement. The underlying cause is the SQLite extended result code 2067:

https://sqlite.org/rescode.html
(2067) SQLITE_CONSTRAINT_UNIQUE
The SQLITE_CONSTRAINT_UNIQUE error code is an extended error code for SQLITE_CONSTRAINT indicating that a UNIQUE constraint failed.

addGeometryColumnsEntry is called twice with these values:

[Seven, pointProperty, Point, 32615, 0, 0]
[Seven, boundedBy, Polygon, 32615, 0, 0]

but gpkg_geometry_columns.sql has the following constraint that limits each table to one geometry column:

CONSTRAINT uk_gc_table_name UNIQUE (table_name)

Kind regards,
Ben.

On 27/04/17 14:00, Ben Caradoc-Davies wrote:

Torben,

I also see these failures locally on master:

Failed tests:
testTileEntries(org.geoserver.geopkg.GeoPackageGetMapOutputFormatTest)
Tests in error:
testGetFeatureWithFilter(org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest):
Error occurred creating table
testGetFeatureTwoTypes(org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest):
Error occurred creating table

The root cause is "java.sql.SQLException: The prepared statement has
been finalized".

The first failure has the following stack trace :

Running org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest
Tests run: 5, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 4.402
sec <<< FAILURE!
testGetFeatureWithFilter(org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest)
  Time elapsed: 3505 sec <<< ERROR!
java.io.IOException: Error occurred creating table
  at org.geotools.jdbc.JDBCDataStore.createSchema(JDBCDataStore.java:800)
  at org.geotools.geopkg.GeoPackage.create(GeoPackage.java:629)
  at org.geotools.geopkg.GeoPackage.add(GeoPackage.java:654)
  at
org.geoserver.geopkg.GeoPackageGetFeatureOutputFormat.write(GeoPackageGetFeatureOutputFormat.java:110)
  at
org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest.testGetFeature(GeoPackageGetFeatureOutputFormatTest.java:137)
  at
org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest.testGetFeatureWithFilter(GeoPackageGetFeatureOutputFormatTest.java:104)
[... omitting surefile/junit-related frames ...]
Caused by: java.sql.SQLException: java.io.IOException:
java.sql.SQLException: The prepared statement has been finalized
  at
org.geotools.geopkg.GeoPkgDialect.postCreateTable(GeoPkgDialect.java:302)
  at org.geotools.jdbc.JDBCDataStore.createSchema(JDBCDataStore.java:797)
  ... 39 more
Caused by: java.io.IOException: java.sql.SQLException: The prepared
statement has been finalized
  at
org.geotools.geopkg.GeoPackage.addGeometryColumnsEntry(GeoPackage.java:932)
  at
org.geotools.geopkg.GeoPkgDialect.postCreateTable(GeoPkgDialect.java:297)
  ... 40 more
Caused by: java.sql.SQLException: The prepared statement has been finalized
  at org.sqlite.core.NativeDB.throwex(NativeDB.java:429)
  at org.sqlite.core.NativeDB.reset(Native Method)
  at
org.sqlite.jdbc3.JDBC3PreparedStatement.execute(JDBC3PreparedStatement.java:57)
  at
org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
  at
org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
  at
org.geotools.geopkg.GeoPackage.addGeometryColumnsEntry(GeoPackage.java:924)
  ... 41 more

Kind regards,
Ben.

On 27/04/17 11:29, Torben Barsballe wrote:

GeoServer side of the fix (WMS GeoPackage output format) here:
https://github.com/geoserver/geoserver/pull/2285

I still get a couple of GeoPackage tests failing, however they are
unrelated to the change and are also failing on master (and 2.11.x, 2.10.x,
2.9.x ... perhaps this is only failing on my machine?). I'm not having much
luck tracking down the root cause of those failures:

Tests in error:

  testGetFeatureWithFilter(org.geoserver.geopkg.
GeoPackageGetFeatureOutputFormatTest): Error occurred creating table

  testGetFeatureTwoTypes(org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest):
Error occurred creating table

If anyone wants to test master locally, and see if those tests run for them
or are failing, that would be appreciated.

Torben

On Wed, Apr 26, 2017 at 1:17 PM, Torben Barsballe <
tbarsballe@anonymised.com> wrote:

I've discovered some interesting behaviour in the GeoPackage Raster Mosaic
implementation over the last few days.

I tried viewing a GeoPackage generated from the GeoServer GeoPackage WMS
output format in QGIS, and observed that tile order is reversed along the
y-axis.

QGIS used GDAL for parsing GeoPackages. After running this issue by the
GDAL team, I was directed to this line of the OGC GeoPackage spec
<http://www.geopackage.org/spec/#tile_matrix&gt;:

The tile coordinate (0,0) always refers to the tile in the upper left

corner of the tile matrix at any zoom level, regardless of the actual
availability of that tile

Adding some test cases to the GeoTools GeoPackage plugin, I can confirm
that GeoTools gets this order wrong. Among other things, this means that
GeoTools / GeoServer will incorrectly read valid GeoPackage files. A fix
can be found here: https://github.com/geotools/geotools/pull/1567

Moving along to GeoServer, it appears there are similar problems with the
GeoServer GeoPackage Community module, namely in
the GeoPackageGetMapOutputFormat.

I believe I have a fix, but am currently being slowed down by the fact
that the GeoPackage module tests don't seem to be passing on master.

*This issue does bring up a problem*: Because GeoTools / GeoServer is
consistently wrong (a GeoPackage generated by GeoServer will be read in as
"correct"), this bug fix could cause problems for anyone who has solely
been using GeoPackage Rasters that were generated by GeoTools / GeoServer.

For more details:
https://osgeo-org.atlassian.net/browse/GEOT-5715
https://osgeo-org.atlassian.net/browse/GEOS-8112
https://trac.osgeo.org/gdal/ticket/6871

Torben

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

Torben,

git blame indicates that the constraint was added in this commit:
https://github.com/geotools/geotools/commit/4c367628f7ca13d43cebccb4a3ee50cd2d72bb3c

The constraint is straight out of the OGC GeoPackage Encoding Standard: Appendix A: Table Definition SQL (Normative)
https://github.com/opengeospatial/geopackage/blob/master/spec/annexes/ddl.adoc

Removing the constraint fixes GeoPackageGetFeatureOutputFormatTest but GeoPackageGetMapOutputFormatTest still fails:

Running org.geoserver.geopkg.GeoPackageGetMapOutputFormatTest
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.897 sec <<< FAILURE!
testTileEntries(org.geoserver.geopkg.GeoPackageGetMapOutputFormatTest) Time elapsed: 887 sec <<< FAILURE!
java.lang.AssertionError
  at org.junit.Assert.fail(Assert.java:86)
  at org.junit.Assert.assertTrue(Assert.java:41)
  at org.junit.Assert.assertNotNull(Assert.java:621)
  at org.junit.Assert.assertNotNull(Assert.java:631)
  at org.geoserver.geopkg.GeoPackageGetMapOutputFormatTest.testTileEntries(GeoPackageGetMapOutputFormatTest.java:65)

Kind regards,
Ben.

On 27/04/17 14:50, Ben Caradoc-Davies wrote:

Torben,

the "java.sql.SQLException: The prepared statement has been finalized"
is caused by failed error handling trying to double-finalize the
prepared statement. The underlying cause is the SQLite extended result
code 2067:

https://sqlite.org/rescode.html
(2067) SQLITE_CONSTRAINT_UNIQUE
The SQLITE_CONSTRAINT_UNIQUE error code is an extended error code for
SQLITE_CONSTRAINT indicating that a UNIQUE constraint failed.

addGeometryColumnsEntry is called twice with these values:

[Seven, pointProperty, Point, 32615, 0, 0]
[Seven, boundedBy, Polygon, 32615, 0, 0]

but gpkg_geometry_columns.sql has the following constraint that limits
each table to one geometry column:

CONSTRAINT uk_gc_table_name UNIQUE (table_name)

Kind regards,
Ben.

On 27/04/17 14:00, Ben Caradoc-Davies wrote:

Torben,

I also see these failures locally on master:

Failed tests:
testTileEntries(org.geoserver.geopkg.GeoPackageGetMapOutputFormatTest)
Tests in error:
testGetFeatureWithFilter(org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest):
Error occurred creating table
testGetFeatureTwoTypes(org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest):
Error occurred creating table

The root cause is "java.sql.SQLException: The prepared statement has
been finalized".

The first failure has the following stack trace :

Running org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest
Tests run: 5, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 4.402
sec <<< FAILURE!
testGetFeatureWithFilter(org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest)
  Time elapsed: 3505 sec <<< ERROR!
java.io.IOException: Error occurred creating table
  at org.geotools.jdbc.JDBCDataStore.createSchema(JDBCDataStore.java:800)
  at org.geotools.geopkg.GeoPackage.create(GeoPackage.java:629)
  at org.geotools.geopkg.GeoPackage.add(GeoPackage.java:654)
  at
org.geoserver.geopkg.GeoPackageGetFeatureOutputFormat.write(GeoPackageGetFeatureOutputFormat.java:110)
  at
org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest.testGetFeature(GeoPackageGetFeatureOutputFormatTest.java:137)
  at
org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest.testGetFeatureWithFilter(GeoPackageGetFeatureOutputFormatTest.java:104)
[... omitting surefile/junit-related frames ...]
Caused by: java.sql.SQLException: java.io.IOException:
java.sql.SQLException: The prepared statement has been finalized
  at
org.geotools.geopkg.GeoPkgDialect.postCreateTable(GeoPkgDialect.java:302)
  at org.geotools.jdbc.JDBCDataStore.createSchema(JDBCDataStore.java:797)
  ... 39 more
Caused by: java.io.IOException: java.sql.SQLException: The prepared
statement has been finalized
  at
org.geotools.geopkg.GeoPackage.addGeometryColumnsEntry(GeoPackage.java:932)
  at
org.geotools.geopkg.GeoPkgDialect.postCreateTable(GeoPkgDialect.java:297)
  ... 40 more
Caused by: java.sql.SQLException: The prepared statement has been finalized
  at org.sqlite.core.NativeDB.throwex(NativeDB.java:429)
  at org.sqlite.core.NativeDB.reset(Native Method)
  at
org.sqlite.jdbc3.JDBC3PreparedStatement.execute(JDBC3PreparedStatement.java:57)
  at
org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
  at
org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
  at
org.geotools.geopkg.GeoPackage.addGeometryColumnsEntry(GeoPackage.java:924)
  ... 41 more

Kind regards,
Ben.

On 27/04/17 11:29, Torben Barsballe wrote:

GeoServer side of the fix (WMS GeoPackage output format) here:
https://github.com/geoserver/geoserver/pull/2285

I still get a couple of GeoPackage tests failing, however they are
unrelated to the change and are also failing on master (and 2.11.x, 2.10.x,
2.9.x ... perhaps this is only failing on my machine?). I'm not having much
luck tracking down the root cause of those failures:

Tests in error:

  testGetFeatureWithFilter(org.geoserver.geopkg.
GeoPackageGetFeatureOutputFormatTest): Error occurred creating table

  testGetFeatureTwoTypes(org.geoserver.geopkg.GeoPackageGetFeatureOutputFormatTest):
Error occurred creating table

If anyone wants to test master locally, and see if those tests run for them
or are failing, that would be appreciated.

Torben

On Wed, Apr 26, 2017 at 1:17 PM, Torben Barsballe <
tbarsballe@anonymised.com> wrote:

I've discovered some interesting behaviour in the GeoPackage Raster Mosaic
implementation over the last few days.

I tried viewing a GeoPackage generated from the GeoServer GeoPackage WMS
output format in QGIS, and observed that tile order is reversed along the
y-axis.

QGIS used GDAL for parsing GeoPackages. After running this issue by the
GDAL team, I was directed to this line of the OGC GeoPackage spec
<http://www.geopackage.org/spec/#tile_matrix&gt;:

The tile coordinate (0,0) always refers to the tile in the upper left

corner of the tile matrix at any zoom level, regardless of the actual
availability of that tile

Adding some test cases to the GeoTools GeoPackage plugin, I can confirm
that GeoTools gets this order wrong. Among other things, this means that
GeoTools / GeoServer will incorrectly read valid GeoPackage files. A fix
can be found here: https://github.com/geotools/geotools/pull/1567

Moving along to GeoServer, it appears there are similar problems with the
GeoServer GeoPackage Community module, namely in
the GeoPackageGetMapOutputFormat.

I believe I have a fix, but am currently being slowed down by the fact
that the GeoPackage module tests don't seem to be passing on master.

*This issue does bring up a problem*: Because GeoTools / GeoServer is
consistently wrong (a GeoPackage generated by GeoServer will be read in as
"correct"), this bug fix could cause problems for anyone who has solely
been using GeoPackage Rasters that were generated by GeoTools / GeoServer.

For more details:
https://osgeo-org.atlassian.net/browse/GEOT-5715
https://osgeo-org.atlassian.net/browse/GEOS-8112
https://trac.osgeo.org/gdal/ticket/6871

Torben

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

Torben,

the last failure is caused by your addition of the constant WORLD ("world" with a lowercase "w") to WMSTestSupport five days ago:
https://github.com/geoserver/geoserver/commit/240983e44e692f453a3d8565a0387b896d73ac2d

This new constant inherited by GeoPackageGetMapOutputFormatTest masks the constant WORLD ("World" with an uppercase "W")from MockData referenced in the static import MockData.* at the top of GeoPackageGetMapOutputFormatTest:
https://github.com/geoserver/geoserver/blob/master/src/main/src/test/java/org/geoserver/data/test/MockData.java#L204

You can make the test pass by changing "World_Lakes" to "world_Lakes":
https://github.com/geoserver/geoserver/blob/master/src/community/geopkg/src/test/java/org/geoserver/geopkg/GeoPackageGetMapOutputFormatTest.java#L65

but you should probably use MockData.WORLD instead of WMSTestSupport.WORLD in WMSTestSupport to reduce duplication, standardise names, and fix GeoPackageGetMapOutputFormatTest. WMSTestSupport already imports MockData.

Kind regards,
Ben.

On 27/04/17 15:05, Ben Caradoc-Davies wrote:

GeoPackageGetMapOutputFormatTest still fails:

Running org.geoserver.geopkg.GeoPackageGetMapOutputFormatTest
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.897
sec <<< FAILURE!
testTileEntries(org.geoserver.geopkg.GeoPackageGetMapOutputFormatTest)
Time elapsed: 887 sec <<< FAILURE!
java.lang.AssertionError
  at org.junit.Assert.fail(Assert.java:86)
  at org.junit.Assert.assertTrue(Assert.java:41)
  at org.junit.Assert.assertNotNull(Assert.java:621)
  at org.junit.Assert.assertNotNull(Assert.java:631)
  at
org.geoserver.geopkg.GeoPackageGetMapOutputFormatTest.testTileEntries(GeoPackageGetMapOutputFormatTest.java:65)

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

Torben,

the underlying problem for GeoPackageGetFeatureOutputFormatTest is that SystemTestData.SEVEN has two geometry columns and this violates the GeoPackage spec:

"Requirement 30: A feature table SHALL have only one geometry column."
https://github.com/opengeospatial/geopackage/blob/master/spec/2a_features.adoc

The SQL constraint matches the GeoPackage spec.

Kind regards,
Ben.

On 27/04/17 15:05, Ben Caradoc-Davies wrote:

git blame indicates that the constraint was added in this commit:
https://github.com/geotools/geotools/commit/4c367628f7ca13d43cebccb4a3ee50cd2d72bb3c
The constraint is straight out of the OGC GeoPackage Encoding Standard:
Appendix A: Table Definition SQL (Normative)
https://github.com/opengeospatial/geopackage/blob/master/spec/annexes/ddl.adoc
Removing the constraint fixes GeoPackageGetFeatureOutputFormatTest

[...]

On 27/04/17 14:50, Ben Caradoc-Davies wrote:

The underlying cause is the SQLite extended result
code 2067:
https://sqlite.org/rescode.html
(2067) SQLITE_CONSTRAINT_UNIQUE
The SQLITE_CONSTRAINT_UNIQUE error code is an extended error code for
SQLITE_CONSTRAINT indicating that a UNIQUE constraint failed.
addGeometryColumnsEntry is called twice with these values:
[Seven, pointProperty, Point, 32615, 0, 0]
[Seven, boundedBy, Polygon, 32615, 0, 0]
but gpkg_geometry_columns.sql has the following constraint that limits
each table to one geometry column:
CONSTRAINT uk_gc_table_name UNIQUE (table_name)

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

Hi Ben,

Thanks for tracking down all of this!

I’ve updated my PR accordingly, and tests now pass locally. Swapped SEVEN and FIFTEEN for LAKES and STREAMS to get the GeoPackage feature tests working.

Torben

···

On Wed, Apr 26, 2017 at 10:47 PM, Ben Caradoc-Davies <ben@anonymised.com> wrote:

Torben,

the underlying problem for GeoPackageGetFeatureOutputFormatTest is that SystemTestData.SEVEN has two geometry columns and this violates the GeoPackage spec:

“Requirement 30: A feature table SHALL have only one geometry column.”
https://github.com/opengeospatial/geopackage/blob/master/spec/2a_features.adoc

The SQL constraint matches the GeoPackage spec.

Kind regards,
Ben.

On 27/04/17 15:05, Ben Caradoc-Davies wrote:

git blame indicates that the constraint was added in this commit:
https://github.com/geotools/geotools/commit/4c367628f7ca13d43cebccb4a3ee50cd2d72bb3c
The constraint is straight out of the OGC GeoPackage Encoding Standard:
Appendix A: Table Definition SQL (Normative)
https://github.com/opengeospatial/geopackage/blob/master/spec/annexes/ddl.adoc
Removing the constraint fixes GeoPackageGetFeatureOutputFormatTest

[…]

On 27/04/17 14:50, Ben Caradoc-Davies wrote:

The underlying cause is the SQLite extended result
code 2067:
https://sqlite.org/rescode.html
(2067) SQLITE_CONSTRAINT_UNIQUE
The SQLITE_CONSTRAINT_UNIQUE error code is an extended error code for
SQLITE_CONSTRAINT indicating that a UNIQUE constraint failed.
addGeometryColumnsEntry is called twice with these values:
[Seven, pointProperty, Point, 32615, 0, 0]
[Seven, boundedBy, Polygon, 32615, 0, 0]
but gpkg_geometry_columns.sql has the following constraint that limits
each table to one geometry column:
CONSTRAINT uk_gc_table_name UNIQUE (table_name)


Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/>
New Zealand

Any good idea on how to detect and fix existing files? How many existing files will be out there?

···

On 26 April 2017 at 13:17, Torben Barsballe <tbarsballe@anonymised.com> wrote:

I’ve discovered some interesting behaviour in the GeoPackage Raster Mosaic implementation over the last few days.

I tried viewing a GeoPackage generated from the GeoServer GeoPackage WMS output format in QGIS, and observed that tile order is reversed along the y-axis.

QGIS used GDAL for parsing GeoPackages. After running this issue by the GDAL team, I was directed to this line of the OGC GeoPackage spec:

The tile coordinate (0,0) always refers to the tile in the upper left corner of the tile matrix at any zoom level, regardless of the actual availability of that tile

Adding some test cases to the GeoTools GeoPackage plugin, I can confirm that GeoTools gets this order wrong. Among other things, this means that GeoTools / GeoServer will incorrectly read valid GeoPackage files. A fix can be found here: https://github.com/geotools/geotools/pull/1567

Moving along to GeoServer, it appears there are similar problems with the GeoServer GeoPackage Community module, namely in the GeoPackageGetMapOutputFormat.

I believe I have a fix, but am currently being slowed down by the fact that the GeoPackage module tests don’t seem to be passing on master.

This issue does bring up a problem: Because GeoTools / GeoServer is consistently wrong (a GeoPackage generated by GeoServer will be read in as “correct”), this bug fix could cause problems for anyone who has solely been using GeoPackage Rasters that were generated by GeoTools / GeoServer.

For more details:
https://osgeo-org.atlassian.net/browse/GEOT-5715
https://osgeo-org.atlassian.net/browse/GEOS-8112
https://trac.osgeo.org/gdal/ticket/6871

Torben


Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


GeoTools-Devel mailing list
GeoTools-Devel@anonymised.com66…sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel


Jody Garnett

There is no easy automated way of detecting existing files, since the files are technically valid, the tiles are just in the wrong order. Existing files will be quite apparent from visual inspection however.

How many existing files are out there depends how many people have been using the GeoServer GeoPackage WMS output format, as that is the only way to generate these files.

Torben

···

On Thu, Apr 27, 2017 at 1:39 PM, Jody Garnett <jody.garnett@anonymised.com> wrote:

Any good idea on how to detect and fix existing files? How many existing files will be out there?


Jody Garnett

On 26 April 2017 at 13:17, Torben Barsballe <tbarsballe@anonymised.com> wrote:

I’ve discovered some interesting behaviour in the GeoPackage Raster Mosaic implementation over the last few days.

I tried viewing a GeoPackage generated from the GeoServer GeoPackage WMS output format in QGIS, and observed that tile order is reversed along the y-axis.

QGIS used GDAL for parsing GeoPackages. After running this issue by the GDAL team, I was directed to this line of the OGC GeoPackage spec:

The tile coordinate (0,0) always refers to the tile in the upper left corner of the tile matrix at any zoom level, regardless of the actual availability of that tile

Adding some test cases to the GeoTools GeoPackage plugin, I can confirm that GeoTools gets this order wrong. Among other things, this means that GeoTools / GeoServer will incorrectly read valid GeoPackage files. A fix can be found here: https://github.com/geotools/geotools/pull/1567

Moving along to GeoServer, it appears there are similar problems with the GeoServer GeoPackage Community module, namely in the GeoPackageGetMapOutputFormat.

I believe I have a fix, but am currently being slowed down by the fact that the GeoPackage module tests don’t seem to be passing on master.

This issue does bring up a problem: Because GeoTools / GeoServer is consistently wrong (a GeoPackage generated by GeoServer will be read in as “correct”), this bug fix could cause problems for anyone who has solely been using GeoPackage Rasters that were generated by GeoTools / GeoServer.

For more details:
https://osgeo-org.atlassian.net/browse/GEOT-5715
https://osgeo-org.atlassian.net/browse/GEOS-8112
https://trac.osgeo.org/gdal/ticket/6871

Torben


Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


GeoTools-Devel mailing list
GeoTools-Devel@anonymised.comge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel