[Geoserver-users] GeoServer without PostGIS, is it a good idea?

Dear GeoServer users,

my company develops systems in meteorology, climatology, radiation
monitoring etc. It's mostly data collection, storing and later retrieval.
Our customers started to demand maps, recently.

What we need is a map client serving customers' data, both rasters (e.g.
forecasts from GRIB files) and vector (e.g. various values measured by
stations from a national climatology network). We looked out for some
options and settled on a combination of a GeoServer & OpenLayers
integrated into our existing web interface.

After several weeks of playing with GeoServer I got an impression that
GeoServer works great in combination with a local PostGIS instance.
PostGIS fits well into our scenario, too: it would contain metadata about
our rasters with time dimensions, and whole vector layers.

The problem is we already have our own infrastructure which does all the
bookkeeping of all our datasets. E.g. we already store our raster data
somewhere in the filesystem and there is a table in our database which
tracks metadata (origin, time, etc.) about these rasters. For climatology
data, we use Oracle database and have our own schema which stores
information about weather stations (e.g. name, location) and current and
historical measured values.

Because of this, my manager does not want to allow me to create a PostGIS
instance along with a GeoServer instance. His arguments is it's
unnecessary duplicity, it won't be consistent with our database unless we
invest heavily in syncing the two and it needs additional maintenance. His
original idea was that our system will provide data to a GeoServer
instance via some Web Service / WFS / whatever and GeoServer will just
"render" it. The problem is our system does not have a WFS interface
(yet?) and our existing interfaces are not OGC standardized. Of course, if
we implement WFS we won't need GeoServer (mostly).

I tried to explain to him that we should treat the PostGIS store as some
kind of internal GeoServer store (just like various index files on disk -
he is a database guy and does not care about all those files created by
GeoServer - he cares about databases) and not try to mess with it in order
to speed up development (deadlines are tight, as usual).

I need either better arguments in this debate or an alternative solution.

I stumbled upon DataStore Development tutorial on GeoTools (
http://docs.geotools.org/latest/tutorials/advanced/datastore.html ) which
looks promising, but I'm not quite sure if it's the right thing. I'll try
to implement a proof-of-concept to see if I'm able to convert our existing
database schema into a new Data Store type recognized by GeoSever.

In the meantime I'd love to hear your opinion about the case. Is it common
to implement custom DataStores from existing databases or mirroring it in
PostGIS is the preferred way? Are there any other options?

--
Peter Kovac
IMS Programmer
MicroStep-MIS
peter.kovac@anonymised.com

Hi Peter,

GeoServer has data drivers for both MS SQL Server and Oracle, but I believe you have to install them separately as extensions from the download page.
MS SQL Server: http://sourceforge.net/projects/geoserver/files/GeoServer/2.7.1/extensions/geoserver-2.7.1-sqlserver-plugin.zip
Oracle: http://sourceforge.net/projects/geoserver/files/GeoServer/2.7.1/extensions/geoserver-2.7.1-oracle-plugin.zip

Once you have your appropriate extension installed, you can use them to either serve the native tables, native views, or add an SQL View to the GeoServer side on the "New Layer" screen from the " On databases you can also create a new feature type by configuring a native SQL statement. Configure new SQL view..."

This allows you to define a relatively simple SQL statement that GeoServer will execute against the backend for WMS/WFS results.

Hope this helps.

Chris Snider
Senior Software Engineer
Intelligent Software Solutions, Inc.

-----Original Message-----
From: Peter Kovac [mailto:peter.kovac@anonymised.com]
Sent: Wednesday, June 24, 2015 5:24 AM
To: geoserver-users@lists.sourceforge.net
Subject: [Geoserver-users] GeoServer without PostGIS, is it a good idea?

Dear GeoServer users,

my company develops systems in meteorology, climatology, radiation monitoring etc. It's mostly data collection, storing and later retrieval.
Our customers started to demand maps, recently.

What we need is a map client serving customers' data, both rasters (e.g.
forecasts from GRIB files) and vector (e.g. various values measured by stations from a national climatology network). We looked out for some options and settled on a combination of a GeoServer & OpenLayers integrated into our existing web interface.

After several weeks of playing with GeoServer I got an impression that GeoServer works great in combination with a local PostGIS instance.
PostGIS fits well into our scenario, too: it would contain metadata about our rasters with time dimensions, and whole vector layers.

The problem is we already have our own infrastructure which does all the bookkeeping of all our datasets. E.g. we already store our raster data somewhere in the filesystem and there is a table in our database which tracks metadata (origin, time, etc.) about these rasters. For climatology data, we use Oracle database and have our own schema which stores information about weather stations (e.g. name, location) and current and historical measured values.

Because of this, my manager does not want to allow me to create a PostGIS instance along with a GeoServer instance. His arguments is it's unnecessary duplicity, it won't be consistent with our database unless we invest heavily in syncing the two and it needs additional maintenance. His original idea was that our system will provide data to a GeoServer instance via some Web Service / WFS / whatever and GeoServer will just "render" it. The problem is our system does not have a WFS interface
(yet?) and our existing interfaces are not OGC standardized. Of course, if we implement WFS we won't need GeoServer (mostly).

I tried to explain to him that we should treat the PostGIS store as some kind of internal GeoServer store (just like various index files on disk - he is a database guy and does not care about all those files created by GeoServer - he cares about databases) and not try to mess with it in order to speed up development (deadlines are tight, as usual).

I need either better arguments in this debate or an alternative solution.

I stumbled upon DataStore Development tutorial on GeoTools ( http://docs.geotools.org/latest/tutorials/advanced/datastore.html ) which looks promising, but I'm not quite sure if it's the right thing. I'll try to implement a proof-of-concept to see if I'm able to convert our existing database schema into a new Data Store type recognized by GeoSever.

In the meantime I'd love to hear your opinion about the case. Is it common to implement custom DataStores from existing databases or mirroring it in PostGIS is the preferred way? Are there any other options?

--
Peter Kovac
IMS Programmer
MicroStep-MIS
peter.kovac@anonymised.com

------------------------------------------------------------------------------
Monitor 25 network devices or servers for free with OpManager!
OpManager is web-based network management software that monitors
network devices and physical & virtual servers, alerts via email & sms
for fault. Monitor 25 devices for free with no restriction. Download now
http://ad.doubleclick.net/ddm/clk/292181274;119417398;o
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

We do have a small POSTGis instance, used largely for "project level" databases as opposed to "corporate level", but most of what we serve through geoserver is coming from the corporate Oracle database.

Notice: This email and any attachments are confidential.
If received in error please destroy and immediately notify us.
Do not copy or disclose the contents.

Peter,

your proposal to use a PostGIS instance on the same server as GeoServer is what I know as the Forward Cache Pattern:
https://www.google.co.nz/search?q=forward%20cache%20pattern
http://flylib.com/books/en/2.79.1.86/1/

Is your spatial data service read-only? How fresh does the data need to be? A forward cache is easiest to implement with read-only data where real-time updates are not required.

In my experience, a forward cache is a great way to improve the quality of data from disparate sources. I always choose PostGIS as I have found it very easy to administer and it seems to run unattended without the regular human sacrifice required by Oracle.

Advantages of a PostGIS forward cache:

- Performance: local PostGIS is very fast, and it is much easier to diagnose and fix performance bottlenecks in the forward cache rather than in a network of sources
- Indices: PostGIS can calculate indices for data sources that do not support them
- Constraints: PostGIS can be configured to reject inconsistent data, for example, data in a shapefile may reference a feature in a database that no longer exists; a PostGIS forward cache constraint will tell you
- Relationships: views/joins that calculate relationships between types from different sources; you can also deliver complex types with app-schema to make these relationships available to users
- Security isolation, as scheduled tasks can push data into PostGIS in a DMZ server, with no need for incoming connections into your internal network
- Confidentiality: data loading process can select only the subset approved for publication, reducing the risk of unwanted disclosures
- Redundancy: GeoServer can keep on using PostGIS even when the Oracle or remote filesystems are down

Yes, there is an additional cost in implementing a forward cache, but the benefits are substantial. I have seen several examples where insistence on using a bare corporate database (driven by managerial and DBA pressure) has resulted in poor outcomes for web service end users.

If you are not permitted to use a forward cache and need to access your non-OGC back end web services, you might want to consider the unsupported webservice data store. The webservice data store was written to allow GeoServer to consume arbitrary SOAP web services as a back end for app-schema complex feature data stores (usual GeoServer OGC front end):
https://www.seegrid.csiro.au/wiki/Infosrvices/GeoserverWebserviceUserGuide
https://github.com/geotools/geotools/tree/master/modules/unsupported/app-schema/webservice
https://github.com/geoserver/geoserver/tree/master/src/community/app-schema

I do not know if there are any current users of the webservice data store. It is unsupported. You can run the unit tests by building GeoServer with the -Pwebservice profile.

Kind regards,
Ben.

On 24/06/15 23:23, Peter Kovac wrote:

Dear GeoServer users,

my company develops systems in meteorology, climatology, radiation
monitoring etc. It's mostly data collection, storing and later retrieval.
Our customers started to demand maps, recently.

What we need is a map client serving customers' data, both rasters (e.g.
forecasts from GRIB files) and vector (e.g. various values measured by
stations from a national climatology network). We looked out for some
options and settled on a combination of a GeoServer & OpenLayers
integrated into our existing web interface.

After several weeks of playing with GeoServer I got an impression that
GeoServer works great in combination with a local PostGIS instance.
PostGIS fits well into our scenario, too: it would contain metadata about
our rasters with time dimensions, and whole vector layers.

The problem is we already have our own infrastructure which does all the
bookkeeping of all our datasets. E.g. we already store our raster data
somewhere in the filesystem and there is a table in our database which
tracks metadata (origin, time, etc.) about these rasters. For climatology
data, we use Oracle database and have our own schema which stores
information about weather stations (e.g. name, location) and current and
historical measured values.

Because of this, my manager does not want to allow me to create a PostGIS
instance along with a GeoServer instance. His arguments is it's
unnecessary duplicity, it won't be consistent with our database unless we
invest heavily in syncing the two and it needs additional maintenance. His
original idea was that our system will provide data to a GeoServer
instance via some Web Service / WFS / whatever and GeoServer will just
"render" it. The problem is our system does not have a WFS interface
(yet?) and our existing interfaces are not OGC standardized. Of course, if
we implement WFS we won't need GeoServer (mostly).

I tried to explain to him that we should treat the PostGIS store as some
kind of internal GeoServer store (just like various index files on disk -
he is a database guy and does not care about all those files created by
GeoServer - he cares about databases) and not try to mess with it in order
to speed up development (deadlines are tight, as usual).

I need either better arguments in this debate or an alternative solution.

I stumbled upon DataStore Development tutorial on GeoTools (
http://docs.geotools.org/latest/tutorials/advanced/datastore.html ) which
looks promising, but I'm not quite sure if it's the right thing. I'll try
to implement a proof-of-concept to see if I'm able to convert our existing
database schema into a new Data Store type recognized by GeoSever.

In the meantime I'd love to hear your opinion about the case. Is it common
to implement custom DataStores from existing databases or mirroring it in
PostGIS is the preferred way? Are there any other options?

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

Hello, Peter.

My own personal experience …

We started off connecting Geoserver to MS SQL Server tables to deliver KML to Google Earth. This was because SQL Server was a corporate direction, and consequently our data warehouse sits in SQL Server. I was aware that performance was worse than that of PostGIS but persisted for while, but then we had an upgrade to 64-bit SQL Server and performance halved. I then got approval to install PostGIS on the same servers that were running Geoserver. We use FME to replicate data from SQL Server to PostGIS on a nightly basis, but we are copying only the data that is required (eg. attributes required for styling and filtering) into PostGIS - so we are really just using PostGIS to hold simple ‘layers’ (in GIS-speak) rather than for a relational data structure. In some ways conceptually we are using PostGIS as an alternative to holding layers in shapefiles. When I explained it that way, the database people were happy.

The advantage of replicating the data into the Geoserver/PostGIS system, is that you can trim it right down - it is lean and fit for purpose - so you should get good performance.

Our systems have been running for about 5 years, and we never have to do anything much with PostGIS.

Regards,

David Collins

Senior Geoscientist

···

On Wed, Jun 24, 2015 at 9:23 PM, Peter Kovac <peter.kovac@anonymised.com> wrote:

Dear GeoServer users,

my company develops systems in meteorology, climatology, radiation
monitoring etc. It’s mostly data collection, storing and later retrieval.
Our customers started to demand maps, recently.

What we need is a map client serving customers’ data, both rasters (e.g.
forecasts from GRIB files) and vector (e.g. various values measured by
stations from a national climatology network). We looked out for some
options and settled on a combination of a GeoServer & OpenLayers
integrated into our existing web interface.

After several weeks of playing with GeoServer I got an impression that
GeoServer works great in combination with a local PostGIS instance.
PostGIS fits well into our scenario, too: it would contain metadata about
our rasters with time dimensions, and whole vector layers.

The problem is we already have our own infrastructure which does all the
bookkeeping of all our datasets. E.g. we already store our raster data
somewhere in the filesystem and there is a table in our database which
tracks metadata (origin, time, etc.) about these rasters. For climatology
data, we use Oracle database and have our own schema which stores
information about weather stations (e.g. name, location) and current and
historical measured values.

Because of this, my manager does not want to allow me to create a PostGIS
instance along with a GeoServer instance. His arguments is it’s
unnecessary duplicity, it won’t be consistent with our database unless we
invest heavily in syncing the two and it needs additional maintenance. His
original idea was that our system will provide data to a GeoServer
instance via some Web Service / WFS / whatever and GeoServer will just
“render” it. The problem is our system does not have a WFS interface
(yet?) and our existing interfaces are not OGC standardized. Of course, if
we implement WFS we won’t need GeoServer (mostly).

I tried to explain to him that we should treat the PostGIS store as some
kind of internal GeoServer store (just like various index files on disk -
he is a database guy and does not care about all those files created by
GeoServer - he cares about databases) and not try to mess with it in order
to speed up development (deadlines are tight, as usual).

I need either better arguments in this debate or an alternative solution.

I stumbled upon DataStore Development tutorial on GeoTools (
http://docs.geotools.org/latest/tutorials/advanced/datastore.html ) which
looks promising, but I’m not quite sure if it’s the right thing. I’ll try
to implement a proof-of-concept to see if I’m able to convert our existing
database schema into a new Data Store type recognized by GeoSever.

In the meantime I’d love to hear your opinion about the case. Is it common
to implement custom DataStores from existing databases or mirroring it in
PostGIS is the preferred way? Are there any other options?


Peter Kovac
IMS Programmer
MicroStep-MIS
peter.kovac@anonymised.com


Monitor 25 network devices or servers for free with OpManager!
OpManager is web-based network management software that monitors
network devices and physical & virtual servers, alerts via email & sms
for fault. Monitor 25 devices for free with no restriction. Download now
http://ad.doubleclick.net/ddm/clk/292181274;119417398;o


Geoserver-users mailing list
Geoserver-users@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users