[Geoserver-users] Technical question: how to manage a large amount of data?

Geoserver is amazing.
This should be clear as a first thing.
In a day of work I've installed the software, some sample data in
various formats like shapefile and oracle spatial, and I've published
in a number of clients with WMS/WFS/GeoRSS and more...
Through WFS and OpenLayers i have a powerful web platform and Through
WFS and uDig I'm able to edit my data via http: both of this stuff is
simply amazing.
But in my company we need to publish a very large amount of data (10gb
of vector data in Oracle Spatial and possibly 80gb of raster data in
any format we like): I've created a large dataset (300mb shapefile
loaded in Oracle Spatial) and in a not so bad server machine the
performance are too slow... I don't know if in the workd are available
WFS (+ editing) servers that are faster, but i hope that the geoserver
community know some tricks to speed-up my application... :slight_smile:
Thanks in any case for this great piece of software.
Diego Guidi

P.S: Sorry for the XPOST

On Thu, December 20, 2007 5:05 pm, Diego Guidi wrote:

Geoserver is amazing.
This should be clear as a first thing.
In a day of work I've installed the software, some sample data in
various formats like shapefile and oracle spatial, and I've published in a
number of clients with WMS/WFS/GeoRSS and more... Through WFS and
OpenLayers i have a powerful web platform and Through
WFS and uDig I'm able to edit my data via http: both of this stuff is
simply amazing. But in my company we need to publish a very large amount of
data (10gb of vector data in Oracle Spatial and possibly 80gb of raster
data in any format we like): I've created a large dataset (300mb shapefile
loaded in Oracle Spatial) and in a not so bad server machine the
performance are too slow... I don't know if in the workd are available WFS
(+ editing) servers that are faster, but i hope that the geoserver
community know some tricks to speed-up my application... :slight_smile: Thanks in any
case for this great piece of software. Diego Guidi

Things that are free to try and relatively easy:
* upgrade to geoserver 1.6
* ugprade java to java 6
* follow every suggestion in the "geoserver in production" page on the wiki

Not enough fast? Ok, the suggestion becomes then:
* switch to postgis or
* become a geotools developer and optimize the oracle datastore or
* pay some geotools developer to optimize the oracle datastore

I know this sounds kinda like a joke, but it's not. No one of the devels
is working on the oracle data store, we're just keeping it alive (avoid
major breaks when some other parts of the API changes).

Cheers
Andrea

The other suggestion that can get you much better performance is to use TileCache. If you google TileCache Tutorial you should get a page about how to set it up. We are working on a pure java tilecache, which should eventually hook up to transactions to automatically refresh when edits are done. But for serving wms base layers using tilecache plus openlayers should get you speed increases by sort of ‘cheating’, by taking advantage of caching instead of pure rendering performance.

On 12/20/07, aaime@anonymised.com <aaime@anonymised.com> wrote:

On Thu, December 20, 2007 5:05 pm, Diego Guidi wrote:

Geoserver is amazing.
This should be clear as a first thing.
In a day of work I’ve installed the software, some sample data in
various formats like shapefile and oracle spatial, and I’ve published in a
number of clients with WMS/WFS/GeoRSS and more… Through WFS and
OpenLayers i have a powerful web platform and Through
WFS and uDig I’m able to edit my data via http: both of this stuff is
simply amazing. But in my company we need to publish a very large amount of
data (10gb of vector data in Oracle Spatial and possibly 80gb of raster
data in any format we like): I’ve created a large dataset (300mb shapefile
loaded in Oracle Spatial) and in a not so bad server machine the
performance are too slow… I don’t know if in the workd are available WFS
(+ editing) servers that are faster, but i hope that the geoserver
community know some tricks to speed-up my application… :slight_smile: Thanks in any
case for this great piece of software. Diego Guidi

Things that are free to try and relatively easy:

  • upgrade to geoserver 1.6
  • ugprade java to java 6
  • follow every suggestion in the “geoserver in production” page on the wiki

Not enough fast? Ok, the suggestion becomes then:

  • switch to postgis or
  • become a geotools developer and optimize the oracle datastore or
  • pay some geotools developer to optimize the oracle datastore

I know this sounds kinda like a joke, but it’s not. No one of the devels
is working on the oracle data store, we’re just keeping it alive (avoid
major breaks when some other parts of the API changes).

Cheers
Andrea


This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/


Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

aaime@anonymised.com wrote:

On Thu, December 20, 2007 5:05 pm, Diego Guidi wrote:
  

Geoserver is amazing.
This should be clear as a first thing.
In a day of work I've installed the software, some sample data in
various formats like shapefile and oracle spatial, and I've published in a
number of clients with WMS/WFS/GeoRSS and more... Through WFS and
OpenLayers i have a powerful web platform and Through
WFS and uDig I'm able to edit my data via http: both of this stuff is
simply amazing.

Sweet; we should so quote you on that. Because as open source projects this is one of our goals;
allow people to download and see the software work. Great work all around, and thanks for the
positive reenforcement.

But in my company we need to publish a very large amount of
data (10gb of vector data in Oracle Spatial and possibly 80gb of raster
data in any format we like): I've created a large dataset (300mb shapefile
loaded in Oracle Spatial) and in a not so bad server machine the
performance are too slow... I don't know if in the workd are available WFS
(+ editing) servers that are faster, but i hope that the geoserver
community know some tricks to speed-up my application... :slight_smile: Thanks in any
case for this great piece of software. Diego Guidi
    

I am going to of course ask you the obvious question; how fast is sufficient for your needs. Put together a model of expected and max use and determine the response time you need. For many situations sticking a tile cache in front will help on the rendering side; and I would be surprised if you find that the Transaction performance gets in your way.

I know this sounds kinda like a joke, but it's not. No one of the devels
is working on the oracle data store, we're just keeping it alive (avoid
major breaks when some other parts of the API changes).

As Andrea mentioned; Oracle support suffers a bit due to lack of a paid developer (or dedicated tester). I know Andrea mention hiring someone, you should also know you can cut costs by providing constant testing. Those can be your costs if you are hiring something; or a developers costs in volunteer time. Some tips, especially in the uDig case, it really helps to have your metadata bounds filled out and known; be sure the tables have a spatial index and that primary keys are in order etc...

Andrea did look at various transfer alternatives at one point. Not sure if the benchmarks were published? There is a range of ideas; from good to bad; kicking around on the performance side of things. If you are serving up to Java clients it may be fun to look into the use of "fast info set" (https://fi.dev.java.net/). There are OGC specific things such as BinXML in this direction; but I respect the approach of fast infoset more.

Jody

Just one more point that I am not sure was covered. In addition to the
tips provided on the Geoserver confluence site,
, I have noticed very positive performance enhancements by compiling
and installing the Tomcat native libraries. Explicit details for this
can be found at http://tomcat.apache.org/tomcat-5.5-doc/apr.html. I
beleive on Windows, this is enabled by default.

I am working with similar dataset sizes in both SDE and PostGIS
(although not Oracle spatial) and have experienced much, much quicker
reponses with the combination of TileCache, JAI and the Tomcat Native
libraries.

Best regards,
Jason

On Dec 21, 2007 5:07 AM, Jody Garnett <jgarnett@anonymised.com> wrote:

aaime@anonymised.com wrote:
> On Thu, December 20, 2007 5:05 pm, Diego Guidi wrote:
>
>> Geoserver is amazing.
>> This should be clear as a first thing.
>> In a day of work I've installed the software, some sample data in
>> various formats like shapefile and oracle spatial, and I've published in a
>> number of clients with WMS/WFS/GeoRSS and more... Through WFS and
>> OpenLayers i have a powerful web platform and Through
>> WFS and uDig I'm able to edit my data via http: both of this stuff is
>> simply amazing.
Sweet; we should so quote you on that. Because as open source projects
this is one of our goals;
allow people to download and see the software work. Great work all
around, and thanks for the
positive reenforcement.
>> But in my company we need to publish a very large amount of
>> data (10gb of vector data in Oracle Spatial and possibly 80gb of raster
>> data in any format we like): I've created a large dataset (300mb shapefile
>> loaded in Oracle Spatial) and in a not so bad server machine the
>> performance are too slow... I don't know if in the workd are available WFS
>> (+ editing) servers that are faster, but i hope that the geoserver
>> community know some tricks to speed-up my application... :slight_smile: Thanks in any
>> case for this great piece of software. Diego Guidi
>>
I am going to of course ask you the obvious question; how fast is
sufficient for your needs. Put together a model of expected and max use
and determine the response time you need. For many situations sticking a
tile cache in front will help on the rendering side; and I would be
surprised if you find that the Transaction performance gets in your way.
> I know this sounds kinda like a joke, but it's not. No one of the devels
> is working on the oracle data store, we're just keeping it alive (avoid
> major breaks when some other parts of the API changes).
>
>
As Andrea mentioned; Oracle support suffers a bit due to lack of a paid
developer (or dedicated tester). I know Andrea mention hiring someone,
you should also know you can cut costs by providing constant testing.
Those can be your costs if you are hiring something; or a developers
costs in volunteer time. Some tips, especially in the uDig case, it
really helps to have your metadata bounds filled out and known; be sure
the tables have a spatial index and that primary keys are in order etc...

Andrea did look at various transfer alternatives at one point. Not sure
if the benchmarks were published? There is a range of ideas; from good
to bad; kicking around on the performance side of things. If you are
serving up to Java clients it may be fun to look into the use of "fast
info set" (https://fi.dev.java.net/). There are OGC specific things such
as BinXML in this direction; but I respect the approach of fast infoset
more.

Jody

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users