[Geoserver-devel] environmental variables

Related to Mauro’s request for system property access when writing ftp templates, I would like to set up alternatives to the “data directory” for path definition.

I am not sure if system properties are the best way to manage these kind of environment or instance level configuration settings. For data directory we have a more general approach:

  1. web.xml (web app specific)
  2. system properties (JVM or AppServer specific)
  3. environmental variables (system specific)

For Mauro’s WEB_SITE_URL setting I could see each of these options making sense.

Currently we use data directory for two things: the location of the xml configuration files and as a base URL when working with file formats such as shapefile and geotiff. We have a related of base URL when working with styles and icons.

Customers have long run into the limitation of this approach - setting up an external folder to store large vector and raster datasets and then using absolute file references.

An idea I would like to float (and can write up a proposal if we like it) is to allow an administrator to define additional data locations and assign them a variable name we can use in a data URL.

Examples:

  • ${shared}/annual/zones.shp - example of a customer shared data location defined by admin
  • ${workspace}/data - the default location for data upload, the builtin variable workspace allows for stable URLs when workspace renamed

When using the file browser in addition to data directory, home and file system the locations defined by the administrator can also be listed.

···


Jody Garnett

On Mon, Mar 23, 2015 at 10:27 PM, Jody Garnett <jody.garnett@anonymised.com>
wrote:

Related to Mauro's request for system property access when writing ftp
templates, I would like to set up alternatives to the "data directory" for
path definition.

I am not sure if system properties are the best way to manage these kind
of environment or instance level configuration settings. For data directory
we have a more general approach:

1) web.xml (web app specific)
2) system properties (JVM or AppServer specific)
3) environmental variables (system specific)

For Mauro's WEB_SITE_URL setting I could see each of these options making
sense.

Hum... yeah, those are the 3 ways a setting can be setup in GeoServer, and
GoeServerExtensions.getProperty(String) will do the lookup accordingly.
Might make sense to use that in Mauro's proposal.

Currently we use data directory for two things: the location of the xml
configuration files and as a base URL when working with file formats such
as shapefile and geotiff. We have a related of base URL when working with
styles and icons.

Customers have long run into the limitation of this approach - setting up
an external folder to store large vector and raster datasets and then using
absolute file references.

An idea I would like to float (and can write up a proposal if we like it)
is to allow an administrator to define additional data locations and assign
them a variable name we can use in a data URL.

We actually had that for a long time with the multiple resource loaders (we
just lacked the support for a variable based one) but... didn'nt you kill
that support with the resource store work?

Examples:

* ${shared}/annual/zones.shp - example of a customer shared data location
defined by admin

Where would this variable be set? 1), 2), 3) as above, or somewhere in the
GeoServer global settings?
If they can be changed while GeoServer is running we have to consider it at
the ResourcePool level, all stores using them
must be disposed of.

* ${workspace}/data - the default location for data upload, the builtin
variable workspace allows for stable URLs when workspace renamed

So if you rename the workspace there is going to be some procedure that
moves all the files in that folder to the new one?
Or are you going to invalidate all stores pointing to that variable on
rename?
Plus, it would seem to me that ${workspace} needs to be relative to
something else too, like, I would probably code it up
as ${shared}/${workspace}.

I would also point out that for uploads, we already have a pluggable API in
REST config that can be used to decide where to put the files,
used for that very purpose:
https://github.com/geoserver/geoserver/wiki/GSIP%20114%20-%20PathMapper%20extension%20point%20to%20control%20REST%20file%20upload%20locations

When using the file browser in addition to data directory, home and file
system the locations defined by the administrator can also be listed.

Hum.. .yeah, I guess in theory the browser should be smart enough to
replace the variable when there is a path bit that
uses them?
Or did you want to make it simpler, and just provide the variables as
roots? Works best if a path can contain a single variable I guess...

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

Thanks for the discussion Andrea, comments inline …

···

On 24 March 2015 at 00:24, Andrea Aime <andrea.aime@anonymised.com> wrote:

I did, the facility was not being used (perhaps additional resource loaders could be configured as part of the application context?)

I was thinking somewhere in 1) 2) 3) above - as my concern was configuration portability (the usual story of taking a configuration between test and production environments). I guess the Java EE happy way to do this would be to define the path as a JNDI lookup (much like switching between test and production databases).

I can see how for nodes on a cluster defining as a GeoServer global setting would be appropriate.

Agreed. Right now I just watch all the data stores fail and get disabled when moving content on disk.

Yes. Same deal for workspace styles/icons.

I would like to carefully move the files on disk and kick ResourcePool appropriately.

(Personally my preference is to store large datasets outside of the data directory, this ${workspace} idea is a nod to those who like keeping everything in the data directory and run into trouble when a workspace is renamed.

I was thinking of listing it as a “top level” directory when browsing for data - don’t want this to be too complicated. Since workspace support files are stored in the data directory, this would end up being a short cut for a common (troubling) case :frowning:

Yes, reviewing this as part of the resumable rest API proposal is part of my inspiration.

Just listing them as roots (like data directory, home directory and file system).

Cheers,
Jody


Jody Garnett

An idea I would like to float (and can write up a proposal if we like it) is to allow an administrator to define additional data locations and assign them a variable name we can use in a data URL.

We actually had that for a long time with the multiple resource loaders (we just lacked the support for a variable based one) but… didn’nt you kill that support with the resource store work?

  • ${shared}/annual/zones.shp - example of a customer shared data location defined by admin

Where would this variable be set? 1), 2), 3) as above, or somewhere in the GeoServer global settings?

If they can be changed while GeoServer is running we have to consider it at the ResourcePool level, all stores using them
must be disposed of.

  • ${workspace}/data - the default location for data upload, the builtin variable workspace allows for stable URLs when workspace renamed

So if you rename the workspace there is going to be some procedure that moves all the files in that folder to the new one?

Or are you going to invalidate all stores pointing to that variable on rename?

Plus, it would seem to me that ${workspace} needs to be relative to something else too, like, I would probably code it up
as ${shared}/${workspace}.

I would also point out that for uploads, we already have a pluggable API in REST config that can be used to decide where to put the files, used for that very purpose:
https://github.com/geoserver/geoserver/wiki/GSIP%20114%20-%20PathMapper%20extension%20point%20to%20control%20REST%20file%20upload%20locations

When using the file browser in addition to data directory, home and file system the locations defined by the administrator can also be listed.

Hum… .yeah, I guess in theory the browser should be smart enough to replace the variable when there is a path bit that
uses them?

Or did you want to make it simpler, and just provide the variables as roots? Works best if a path can contain a single variable I guess…

Hi,

···

I like the idea of having custom variables to represent base path for resources. It was already in my radar for geotiffs locations.
I also agree that using GeoServerExtensions.getProperty(…) could be a starting point for both.

Mauro

Examples:

  • ${shared}/annual/zones.shp - example of a customer shared data location defined by admin

Where would this variable be set? 1), 2), 3) as above, or somewhere in the GeoServer global settings?
If they can be changed while GeoServer is running we have to consider it at the ResourcePool level, all stores using them
must be disposed of.

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Dott. Mauro Bartolomeoli
@mauro_bart
Senior Software Engineer

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it


AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.