[Geoserver-users] Re-quest for advice on basic data management strategies in GeoSever

G'Day all,

This is a follow up to an request sent in back in July (http://osgeo-org.1803224.n2.nabble.com/Basic-question-Layer-Style-Strategies-td6609325.html).

Having worked with the GeoServer admin interface for a few months now I'm still baffled to discover the "right" way to manage data layers. Any pointers to relevant documentation or exemplar explanations would be hugely appreciated.

Here's our context:

We've got standalone installs of the latest GeoServer, GeoNetwork, and GeoExplorer all running in the same Tomcat directory, with PostGIS on the back end, and Debian for the OS.

Like many organisations, ours is dominated by ArcGIS. The researchers all run Arc, and have provided their work as shape files. For the most part we are dealing with point and polygon data, with unique style definitions for each layer in the shapefile.

These shapefiles can comprise 100+ layers (each with a distinct legend, abstract, etc).

So, when they send me the shapefile.zip, the first problem is producing an SLD for each of the layers. This is a pretty tedious process in itself.

Once we've got the data and styles loaded up we come back to GeoServer admin.

The shapefile (example.shp) has been loaded once into a data store.

In order to produce the 100+ layers, I have to:

1. Click on Add new resource

2. Choose the datastore

3. Locate example.shp and select [Republish]

4. Give the new layer a unique name, calculate bounding box, manually copy/paste the abstract info

5. Swap to the Publish tab and scroll through several hundred styles to locate the one we're after

6. Click to Save

Now this one layer is added to the growing list of layers defined on the server. Repeat this by 100 times and you have one dataset done. In our case there's about 25 more datasets to go. I'm not going to live through it!

Note that this "republish" technique is great because I don't have to re-upload the shapefile data each time...but 'republish' includes the entire dataset (naturally). So when we look at the map in a viewer and click for feature info, what gets returned is the entire dataset, not just the specific layer that has been styled.

This just doesn't make sense: I'm certain that there's a "management strategy" that I'm not getting....

Hopefully the answer is as simple as some kind soul saying "For goodness sake, would you just read GeoServer 101 (link here)" ... but to date I have not found that resource to address my issues.

Thanks very much in advance for any pointers.

JB

--
John Brisbin
Managing Director, BoaB interactive Pty Ltd
POB 802 Townsville, QLD 4810

M: 0407 471 565 | P: 07 3103 0574
Skype: boabjohn | Twitter: @boabjohn

Hi John,

You may use CURL to upload shapefiles and their respective sld files automatically. Have a look at this link for more info.
http://docs.geoserver.org/stable/en/user/restconfig/rest-config-examples/rest-config-examples-curl.html

Best,
Manouchehr

-----Original Message-----
From: John Brisbin [mailto:boabinteractive@anonymised.com]
Sent: Mittwoch, 7. September 2011 08:51
To: Geoserver-users@lists.sourceforge.net
Subject: [Geoserver-users] Re-quest for advice on basic data management strategies in GeoSever

G'Day all,

This is a follow up to an request sent in back in July (http://osgeo-org.1803224.n2.nabble.com/Basic-question-Layer-Style-Strategies-td6609325.html).

Having worked with the GeoServer admin interface for a few months now I'm still baffled to discover the "right" way to manage data layers. Any pointers to relevant documentation or exemplar explanations would be hugely appreciated.

Here's our context:

We've got standalone installs of the latest GeoServer, GeoNetwork, and GeoExplorer all running in the same Tomcat directory, with PostGIS on the back end, and Debian for the OS.

Like many organisations, ours is dominated by ArcGIS. The researchers all run Arc, and have provided their work as shape files. For the most part we are dealing with point and polygon data, with unique style definitions for each layer in the shapefile.

These shapefiles can comprise 100+ layers (each with a distinct legend, abstract, etc).

So, when they send me the shapefile.zip, the first problem is producing an SLD for each of the layers. This is a pretty tedious process in itself.

Once we've got the data and styles loaded up we come back to GeoServer admin.

The shapefile (example.shp) has been loaded once into a data store.

In order to produce the 100+ layers, I have to:

1. Click on Add new resource

2. Choose the datastore

3. Locate example.shp and select [Republish]

4. Give the new layer a unique name, calculate bounding box, manually copy/paste the abstract info

5. Swap to the Publish tab and scroll through several hundred styles to locate the one we're after

6. Click to Save

Now this one layer is added to the growing list of layers defined on the server. Repeat this by 100 times and you have one dataset done. In our case there's about 25 more datasets to go. I'm not going to live through it!

Note that this "republish" technique is great because I don't have to re-upload the shapefile data each time...but 'republish' includes the entire dataset (naturally). So when we look at the map in a viewer and click for feature info, what gets returned is the entire dataset, not just the specific layer that has been styled.

This just doesn't make sense: I'm certain that there's a "management strategy" that I'm not getting....

Hopefully the answer is as simple as some kind soul saying "For goodness sake, would you just read GeoServer 101 (link here)" ... but to date I have not found that resource to address my issues.

Thanks very much in advance for any pointers.

JB

--
John Brisbin
Managing Director, BoaB interactive Pty Ltd POB 802 Townsville, QLD 4810

M: 0407 471 565 | P: 07 3103 0574
Skype: boabjohn | Twitter: @boabjohn

------------------------------------------------------------------------------
Using storage to extend the benefits of virtualization and iSCSI Virtualization increases hardware utilization and delivers a new level of agility. Learn what those decisions are and how to modernize your storage and backup environments for virtualization.
http://www.accelacomm.com/jaw/sfnl/114/51434361/
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

I haven’t used the rest api but that sounds like what you want. However I wanted to mention two things.

  1. I’d be interested in seeing a “layer copy” feature to be able to quickly setup a new layer that was very similar to an existing one, but change some attributes (including store/layer)

  2. What viewer are you talking about here? I have only used openlayers but you should be able to specify the layer you want info for in a get feature info request.

charles

On Sep 7, 2011, at 2:51 AM, John Brisbin wrote:

Note that this “republish” technique is great because I don’t have to
re-upload the shapefile data each time…but ‘republish’ includes the
entire dataset (naturally). So when we look at the map in a viewer and
click for feature info, what gets returned is the entire dataset, not
just the specific layer that has been styled.

Thanks for the comments Charles.

  1. I think your layer copy feature is almost what you get with [republish] since you can change just about all the attributes in the edit layer dialogs.

  2. The feature data for the whole shapefile is made available when re-publishing the ‘master’ layer. We use OpenLayers as well, and you can see this effect without ever leaving Geoserver as follows:

  • Upload a sample shapefile with layers R, S, T

  • Upload three SLD files one for each layer

  • The shapefile is a “layer” itself…so go to the Layer list and “add new resource”

  • Choose the store/test shapefile and [publish again]

  • Name the layer R

  • Apply the style R

  • save… we now have a new layer R

  • Go to the preview Layers area, choose to preview layer R as OpenLayers WMS

  • A viewer will appear. Click on the map.

  • Feature info for the whole shape file will be presented…not just the data for the styled layer.

This would obviously be the case because at no point did I specify that the data should be limited to the single layer within the “master” shapefile… and this is my point/query: how should this process be done?

I can’t see how the restful cURL addresses this issue. Even if it will automatically create a new layer for each layer in a shapefile (and limit the data to that layer), I can’t see how we associate the SLD file, or assign layer-specific metadata to each layer.

···
John Brisbin
Managing Director, BoaB interactive Pty Ltd
POB 802 Townsville, QLD 4810

M: 0407 471 565 | P: 07 3103 0574
Skype: boabjohn | Twitter: @boabjohn

I haven’t used the rest api but that sounds like what you want. However I wanted to mention two things.

  1. I’d be interested in seeing a “layer copy” feature to be able to quickly setup a new layer that was very similar to an existing one, but change some attributes (including store/layer)

  2. What viewer are you talking about here? I have only used openlayers but you should be able to specify the layer you want info for in a get feature info request.

charles

On Sep 7, 2011, at 2:51 AM, John Brisbin wrote:

Note that this “republish” technique is great because I don’t have to
re-upload the shapefile data each time…but ‘republish’ includes the
entire dataset (naturally). So when we look at the map in a viewer and
click for feature info, what gets returned is the entire dataset, not
just the specific layer that has been styled.

Hi All, is ther a problem with the trunk build right now? I just did a fresh checkout from trunk, then mvn clean install and get the compilation error below.

Thanks for any help you can give me,

Scott

svn co https://svn.codehaus.org/geoserver/trunk geoserver-trunk

Checked out external at revision 16299.

Checked out revision 16299.

                cd geoserver-trunk/src

mvn clean install

[INFO] -------------------------------------------------------------

[ERROR] COMPILATION ERROR :

[INFO] -------------------------------------------------------------

[ERROR] \eclipse_projects\geoserver\geoserver2\geoserver-trunk\src\main\src\main\java\org\geoserver\catalog\ResourcePool.java:[1368,17]

cleaner has private access in org.geotools.util.SoftValueHashMap

[ERROR] \eclipse_projects\geoserver\geoserver2\geoserver-trunk\src\main\src\main\java\org\geoserver\catalog\ResourcePool.java:[1368,47]

<anonymous org.geoserver.catalog.ResourcePool$CatalogResourceCache$1> is not abstract and does not override abstract method clean(java.lang.Object)

in org.geotools.util.SoftValueHashMap.ValueCleaner

[ERROR] \eclipse_projects\geoserver\geoserver2\geoserver-trunk\src\main\src\main\java\org\geoserver\catalog\ResourcePool.java:[1370,16]

method does not override or implement a method from a supertype

[INFO] 3 errors

[INFO] -------------------------------------------------------------

[INFO] ------------------------------------------------------------------------

[INFO] Reactor Summary:

[INFO]

[INFO] GeoServer … SUCCESS [1:34.741s]

[INFO] Core Platform Module … SUCCESS [10.936s]

[INFO] Open Web Service Module … SUCCESS [31.700s]

[INFO] Main Module … FAILURE [1:35.513s]

[INFO] Web Coverage Service Module … SKIPPED

[INFO] Web Coverage Service 1.0 Module … SKIPPED

[INFO] Web Coverage Service 1.1 Module … SKIPPED

[INFO] Web Feature Service Module … SKIPPED

[INFO] Web Map Service Module … SKIPPED

[INFO] GeoWebCache (GWC) Module … SKIPPED

[INFO] REST Support Module … SKIPPED

[INFO] REST Configuration Service Module … SKIPPED

[INFO] GeoServer Web Modules … SKIPPED

[INFO] Core UI Module … SKIPPED

[INFO] WMS UI Module … SKIPPED

[INFO] GWC UI Module … SKIPPED

[INFO] WFS UI Module … SKIPPED

[INFO] Demoes Module … SKIPPED

[INFO] WCS UI Module … SKIPPED

[INFO] Security UI Module … SKIPPED

[INFO] GeoServer Web Application … SKIPPED

[INFO] Community Space … SKIPPED

[INFO] GeoServer Extensions … SKIPPED

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 3:54.827s

[INFO] Finished at: Wed Sep 07 09:13:00 MDT 2011

[INFO] Final Memory: 13M/27M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile (default-compile) on project

main: Compilation failure: Compilation failure:

[ERROR] \eclipse_projects\geoserver\geoserver2\geoserver-trunk\src\main\src\main\java\org\geoserver\catalog\ResourcePool

.java:[1368,17] cleaner has private access in org.geotools.util.SoftValueHashMap

[ERROR] \eclipse_projects\geoserver\geoserver2\geoserver-trunk\src\main\src\main\java\org\geoserver\catalog\ResourcePool

.java:[1368,47] <anonymous org.geoserver.catalog.ResourcePool$CatalogResourceCache$1> is not abstract and does not overr

ide abstract method clean(java.lang.Object) in org.geotools.util.SoftValueHashMap.ValueCleaner

[ERROR] \eclipse_projects\geoserver\geoserver2\geoserver-trunk\src\main\src\main\java\org\geoserver\catalog\ResourcePool

.java:[1370,16] method does not override or implement a method from a supertype

We are considering a product http://www.geocat.net/bridge that might
help you too. I havent tried it yet, but sounds promising at least.

Notice: This email and any attachments are confidential. If received in error please destroy and immediately notify us. Do not copy or disclose the contents.

On Wed, Sep 7, 2011 at 5:27 PM, Scott Ellis <sellis@anonymised.com> wrote:

Hi All, is ther a problem with the trunk build right now? I just did a fresh
checkout from trunk, then mvn clean install and get the compilation error
below.

Thanks for any help you can give me,

We have a continuous integration server building GeoServer once each
hour (assuming a commit has been made) here:
http://hudson.opengeo.org/hudson/

As you can see all builds are fine. The error you're seeing is due to
an API change I performed various days ago in GeoTools and GeoServer.
It would seem you have outdated Geotools jars on your machine.

All active GeoServer developers have an up to date geotools checkout
so we build our own jars, but you're getting them from the repositories
instead. Not sure why, but sometimes it happens that the jars in the repos get
out of synch and we get error reports like this one.
Can you force maven to re-download the snapshot jars from the repos?

Also, I see you're using Maven 3, which afaik forces the usage of timestamped
snapshots, but most people in geoserver land are using maven 2, and the
build servers too, and we dont' have timestamped snapshots, wondering if
that is the cause of the issue

Cheers
Andrea

--
-------------------------------------------------------
Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf

-------------------------------------------------------

When migrating from maven 2 to maven 3 you need to remove from your ~/.m2/repository any SNAPSHOT dependencies - this change got me the same kind of errors you are seeing (out of date jars) when I made the switch…

Jody

On Thu, Sep 8, 2011 at 7:43 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:

On Wed, Sep 7, 2011 at 5:27 PM, Scott Ellis <sellis@anonymised.com> wrote:

Hi All, is ther a problem with the trunk build right now? I just did a fresh
checkout from trunk, then mvn clean install and get the compilation error
below.

Thanks for any help you can give me,

We have a continuous integration server building GeoServer once each
hour (assuming a commit has been made) here:
http://hudson.opengeo.org/hudson/

As you can see all builds are fine. The error you’re seeing is due to
an API change I performed various days ago in GeoTools and GeoServer.
It would seem you have outdated Geotools jars on your machine.

All active GeoServer developers have an up to date geotools checkout
so we build our own jars, but you’re getting them from the repositories
instead. Not sure why, but sometimes it happens that the jars in the repos get
out of synch and we get error reports like this one.
Can you force maven to re-download the snapshot jars from the repos?

Also, I see you’re using Maven 3, which afaik forces the usage of timestamped
snapshots, but most people in geoserver land are using maven 2, and the
build servers too, and we dont’ have timestamped snapshots, wondering if
that is the cause of the issue

Cheers
Andrea

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf



Using storage to extend the benefits of virtualization and iSCSI
Virtualization increases hardware utilization and delivers a new level of
agility. Learn what those decisions are and how to modernize your storage
and backup environments for virtualization.
http://www.accelacomm.com/jaw/sfnl/114/51434361/


Geotools-devel mailing list
Geotools-devel@anonymised.coms.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel