[Geoserver-devel] GEOS-7533 - slow schema creation on postgis with 20k+ tables

Greetings,

This mail is in reference to issue described at https://osgeo-org.atlassian.net/browse/GEOS-7533?oldIssueView=true.

To replicate the user`s environment, I have a Postgis database with 30k tables.

As per Andrea`s initial investigation and feedback on this issue, it is proposed to have the PostGISDialect hold additional Sql types and Sql Name mappings inside the override map.

Secondly, we could have the JDBCDataStore implementation optionally skip going to underlying database if all the dialect implementation has managed to provide all Sql Type Names for given Sql Types.

A draft PR is available at https://github.com/geotools/geotools/pull/2504

regards,
Imran

···

I.R

Hi Imran,

So, if I understand this correctly the issue starts on this method:
https://github.com/geotools/geotools/blob/5f84746a9410d6761056f2d02bc8e340d60959c0/modules/library/jdbc/src/main/java/org/geotools/jdbc/JDBCDataStore.java#L3233-L3343
… and its caused by the way JDBC getTypeInfo() is handled by PostgreSQL, which currently makes this call very inefficient for when we have an high number of tables, did I got this right?

Then yes +1 for your plan, I would just make sure that we log an warning for missing types mappings that we need to look up in the database.

Thank you,

On Mon, 2019-07-22 at 19:25 +0500, Imran Rajjad wrote:

Greetings,

This mail is in reference to issue described at https://osgeo-org.atlassian.net/browse/GEOS-7533?oldIssueView=true.

To replicate the user`s environment, I have a Postgis database with 30k tables.

As per Andrea`s initial investigation and feedback on this issue, it is proposed to have the PostGISDialect hold additional Sql types and Sql Name mappings inside the override map.

Secondly, we could have the JDBCDataStore implementation optionally skip going to underlying database if all the dialect implementation has managed to provide all Sql Type Names for given Sql Types.

A draft PR is available at https://github.com/geotools/geotools/pull/2504

regards,
Imran

_______________________________________________
Geoserver-devel mailing list
[Geoserver-devel@anonymised.comsts.sourceforge.net](mailto:Geoserver-devel@lists.sourceforge.net)
[https://lists.sourceforge.net/lists/listinfo/geoserver-devel](https://lists.sourceforge.net/lists/listinfo/geoserver-devel)

-- 
Regards,
Nuno Oliveira
==
GeoServer Professional Services from the
experts! 
Visit http://goo.gl/it488V for more information.
==

Nuno Miguel Carvalho Oliveira
@nmcoliveira
Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:      +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

Con riferimento alla normativa sul trattamento dei dati 
personali (Reg. UE 2016/679 - Regolamento generale sulla 
protezione dei dati “GDPR”), si precisa che ogni 
circostanza inerente alla presente email (il suo contenuto, 
gli eventuali allegati, etc.) è un dato la cui conoscenza 
è riservata al/i solo/i destinatario/i indicati dallo 
scrivente. Se il messaggio Le è giunto per errore, è 
tenuta/o a cancellarlo, ogni altra operazione è illecita. 
Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to 
which it is addressed and may contain information that 
is privileged, confidential or otherwise protected from 
disclosure. We remind that - as provided by European 
Regulation 2016/679 “GDPR” - copying, dissemination or 
use of this e-mail or the information herein by anyone 
other than the intended recipient is prohibited. If you 
have received this email by mistake, please notify 
us immediately by telephone or e-mail.

Btw, this mail was more suited to GeoTools DEV list :wink:

On Mon, 2019-07-22 at 19:25 +0500, Imran Rajjad wrote:

Greetings,

This mail is in reference to issue described at https://osgeo-org.atlassian.net/browse/GEOS-7533?oldIssueView=true.

To replicate the user`s environment, I have a Postgis database with 30k tables.

As per Andrea`s initial investigation and feedback on this issue, it is proposed to have the PostGISDialect hold additional Sql types and Sql Name mappings inside the override map.

Secondly, we could have the JDBCDataStore implementation optionally skip going to underlying database if all the dialect implementation has managed to provide all Sql Type Names for given Sql Types.

A draft PR is available at https://github.com/geotools/geotools/pull/2504

regards,
Imran

_______________________________________________
Geoserver-devel mailing list
[Geoserver-devel@anonymised.comsts.sourceforge.net](mailto:Geoserver-devel@lists.sourceforge.net)
[https://lists.sourceforge.net/lists/listinfo/geoserver-devel](https://lists.sourceforge.net/lists/listinfo/geoserver-devel)

-- 
Regards,
Nuno Oliveira
==
GeoServer Professional Services from the
experts! 
Visit http://goo.gl/it488V for more information.
==

Nuno Miguel Carvalho Oliveira
@nmcoliveira
Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:      +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

Con riferimento alla normativa sul trattamento dei dati 
personali (Reg. UE 2016/679 - Regolamento generale sulla 
protezione dei dati “GDPR”), si precisa che ogni 
circostanza inerente alla presente email (il suo contenuto, 
gli eventuali allegati, etc.) è un dato la cui conoscenza 
è riservata al/i solo/i destinatario/i indicati dallo 
scrivente. Se il messaggio Le è giunto per errore, è 
tenuta/o a cancellarlo, ogni altra operazione è illecita. 
Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to 
which it is addressed and may contain information that 
is privileged, confidential or otherwise protected from 
disclosure. We remind that - as provided by European 
Regulation 2016/679 “GDPR” - copying, dissemination or 
use of this e-mail or the information herein by anyone 
other than the intended recipient is prohibited. If you 
have received this email by mistake, please notify 
us immediately by telephone or e-mail.