[Geoserver-users] SQL view incorrect not nillable attributes

Hello all,

I have encountered a problem when migrating SQL view layers from an old project using Geoserver 2.14.0 with Oracle to Geoserver 2.22.2 with Postgres. The issue is that Geoserver incorrectly assumes, that some of the attributes in the query are not nillable. This results in a conversion of null values to 0 (for integers), which is not what we want. It affects the styles, that expect null values and also the WMS GetFeatureInfo calls get incorrect data. I tried the new feature type customization option, to force the attributes to be nillable, but it did not help. I have managed to replicate this using a minimal example and it seems to be caused by a query containing a left join, where the table on the right side of the join has "not null" columns. If I remove the "not null" constraint the SQL view works correctly. Did anyone else experience this? Is this a bug?

Following is an example to reproduce the issue:

First we create two tables with some data:
create table table1 (
id integer not null,
value integer not null
);

create table table2 (
id integer not null,
geom geometry(Point, 4326)
);

insert into table1 values (1, 11);
insert into table1 values (2, 22);
insert into table1 values (3, 33);

insert into table2 values (1, ST_MakePoint(1,1));
insert into table2 values (2, ST_MakePoint(2,2));
insert into table2 values (3, ST_MakePoint(3,3));
insert into table2 values (4, ST_MakePoint(4,4));

Here is the SQL query used to create the layer in Geoserver:
select table1.id, table1.value, table2.geom from table2 left join table1 on (table1.id=table2.id)

When the layer is created in Feature type details the properties id and value are incorrectly marked as not nillable. The result can also be tested in layer preview, by clicking on the rightmost point. Expected result is null id and value, but Geoserver returns 0 for both.

best regards,
Gyorgy Tomcsanyi

Hi all,

we did a bit of debugging. The nullability from the SQL view is determined in org.geotools.jdbc.JDBCFeatureSource.getColumnMetadata using the ResultSet.getMetaData().isNullable(i) call. We did some experiments and googling and it looks like this is not reliable for the Postgres JDBC driver. For the scenario below isNullable thinks, that the columns from the left join are not nullable.

In this case we hoped the feature type customization feature could help. Even after setting all columns as nillable Geoserver still converts null values to zeros. Am I correct, that this setting should solve the problem? We might look into this more, could you please point us to the classes that could cause this?

best regards,
Gyorgy Tomcsanyi

···

On 13. 3. 2023 16:28, Gyorgy Tomcsanyi wrote:

Hello all,

I have encountered a problem when migrating SQL view layers from an old project using Geoserver 2.14.0 with Oracle to Geoserver 2.22.2 with Postgres. The issue is that Geoserver incorrectly assumes, that some of the attributes in the query are not nillable. This results in a conversion of null values to 0 (for integers), which is not what we want. It affects the styles, that expect null values and also the WMS GetFeatureInfo calls get incorrect data. I tried the new feature type customization option, to force the attributes to be nillable, but it did not help. I have managed to replicate this using a minimal example and it seems to be caused by a query containing a left join, where the table on the right side of the join has “not null” columns. If I remove the “not null” constraint the SQL view works correctly. Did anyone else experience this? Is this a bug?

Following is an example to reproduce the issue:

First we create two tables with some data:
create table table1 (
id integer not null,
value integer not null
);

create table table2 (
id integer not null,
geom geometry(Point, 4326)
);

insert into table1 values (1, 11);
insert into table1 values (2, 22);
insert into table1 values (3, 33);

insert into table2 values (1, ST_MakePoint(1,1));
insert into table2 values (2, ST_MakePoint(2,2));
insert into table2 values (3, ST_MakePoint(3,3));
insert into table2 values (4, ST_MakePoint(4,4));

Here is the SQL query used to create the layer in Geoserver:
select table1.id, table1.value, table2.geom from table2 left join table1 on (table1.id=table2.id)

When the layer is created in Feature type details the properties id and value are incorrectly marked as not nillable. The result can also be tested in layer preview, by clicking on the rightmost point. Expected result is null id and value, but Geoserver returns 0 for both.

best regards,
Gyorgy Tomcsanyi


Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer

Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

In this case we hoped the feature type customization feature could help. Even after setting all columns as nillable Geoserver still converts null values to zeros. Am I correct, that this setting should solve the problem?

No, it’s happening too late, it applies an in memory transformation on top of data that has been already read from the database
by JDBCDataStore.

We might look into this more, could you please point us to the classes that could cause this?

I don’t see much work arounds besides fixing the JDBC driver itself, there is no way to influence creation of feature types
from outside the JDBCDataStore classes. This is a starting point if you want to do more debugging:

https://github.com/geotools/geotools/blob/main/modules/library/jdbc/src/main/java/org/geotools/jdbc/JDBCFeatureSource.java#L848

Cheers
Andrea

···

==
GeoServer Professional Services from the experts!

Visit http://bit.ly/gs-services-us for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions Group
phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

https://www.geosolutionsgroup.com/

http://twitter.com/geosolutions_it


Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail