[Geoserver-users] Integrated GeoWebCache and DiskQuota on Windows

GeoServer GeoServer User
1

We are attempting to implement caching and are seeing problems with the disk use. The environments we have tested are

GeoServer 2.6, 2.5.x, 2.2.x
Windows 2012 R2, Win 2008 R2
Java 7, Java 6
Tomcat 8, Tomcat 6

The environment does not make any difference - beyond the fact that this is a typical Windows deployment. It fails under all of them.

For cleanup we specified LRU (Least Recently Used). The layers we are caching are dynamic - the data updates each minute (for parts of it). We have both server and client timeouts set on the layer. On the client end, the layers work as expected, and we also see proper behavior for the cache hits (we think). What we don't see is proper cleanup of the cache on the filesystem. There is one oddity, and one major problem. The problem is that the specified size limits are not being honored. For example, I have set the limit to 20GB, but the cache directory ended up with over 100GB of files before I manually cleaned it. The oddity is the size of the H2 DB files. The same system had DB files over 10GB in size. There were no relevant errors in the geoServer logs (set at Production Level logging) during the time it approached and exceeded the limit. The check frequency is set to 5 minutes, but this progression lasted for days. The DiskQuota status size used does NOT match the actual space used on disk (or even close after a while), and also rises above and stays above the set limit.

My first thought is that the system is attempting to delete the old files, but is silently failing. I have made a number of changes to the directory permissions to try to insure that is not a trigger without any improvement.

So I have several questions.

Is the size limit on the Quota page supposed to reflect the layer cache file AND the DB files, or just the former?

Would you normally expect a multi-GB DB file (main and index)?

Is anyone successfully using Disk Quotas with Windows 2008 or later?

If so, are the layers 'dynamic' - i.e. the data changes frequently, and layer timeouts are specified. Could this be contributing to the issue?

Is there a way to 'dump' the quota DB files. I have not used H2, and a quick attempt using its 'Console' does not let me open the DB file.

Does anyone have any suggestions for other ways to approach this (external DB, ...)?

Are there any Java or logging settings I should change to help diagnose this?

Any help would be appreciated.

2

Just curious, is your geowebcache folder under Program Files(x86)?

-----Original Message-----
From: Tom S [mailto:tom-sourceforge@anonymised.com]
Sent: Monday, October 20, 2014 6:01 PM
To: geoserver-users@lists.sourceforge.net
Subject: [Geoserver-users] Integrated GeoWebCache and DiskQuota on Windows

We are attempting to implement caching and are seeing problems with the
disk use. The environments we have tested are

GeoServer 2.6, 2.5.x, 2.2.x
Windows 2012 R2, Win 2008 R2
Java 7, Java 6
Tomcat 8, Tomcat 6

The environment does not make any difference - beyond the fact that this
is a typical Windows deployment. It fails under all of them.

For cleanup we specified LRU (Least Recently Used). The layers we are
caching are dynamic - the data updates each minute (for parts of it).
We have both server and client timeouts set on the layer. On the client
end, the layers work as expected, and we also see proper behavior for
the cache hits (we think). What we don't see is proper cleanup of the
cache on the filesystem. There is one oddity, and one major problem.
The problem is that the specified size limits are not being honored.
For example, I have set the limit to 20GB, but the cache directory ended
up with over 100GB of files before I manually cleaned it. The oddity is
the size of the H2 DB files. The same system had DB files over 10GB in
size. There were no relevant errors in the geoServer logs (set at
Production Level logging) during the time it approached and exceeded the
limit. The check frequency is set to 5 minutes, but this progression
lasted for days. The DiskQuota status size used does NOT match the
actual space used on disk (or even close after a while), and also rises
above and stays above the set limit.

My first thought is that the system is attempting to delete the old
files, but is silently failing. I have made a number of changes to the
directory permissions to try to insure that is not a trigger without any
improvement.

So I have several questions.

Is the size limit on the Quota page supposed to reflect the layer cache
file AND the DB files, or just the former?

Would you normally expect a multi-GB DB file (main and index)?

Is anyone successfully using Disk Quotas with Windows 2008 or later?

If so, are the layers 'dynamic' - i.e. the data changes frequently, and
layer timeouts are specified. Could this be contributing to the issue?

Is there a way to 'dump' the quota DB files. I have not used H2, and a
quick attempt using its 'Console' does not let me open the DB file.

Does anyone have any suggestions for other ways to approach this
(external DB, ...)?

Are there any Java or logging settings I should change to help diagnose
this?

Any help would be appreciated.

------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

3

It is not. Depending on the test environment, I have either relocated the geoserver data dir, or the actual cache dir to a different path (with simple directory names - no spaces, special characters, ....). In all cases the OS, Tomcat, Java are all native 64 bit implementations, so geoserver proper is actually under C:\Program Files\Tomcat....

On 10/21/2014 5:09 AM, Ralph Dell wrote:

Just curious, is your geowebcache folder under Program Files(x86)?

-----Original Message-----
From: Tom S [mailto:tom-sourceforge@anonymised.com]
Sent: Monday, October 20, 2014 6:01 PM
To: geoserver-users@lists.sourceforge.net
Subject: [Geoserver-users] Integrated GeoWebCache and DiskQuota on Windows

4

On Tue, Oct 21, 2014 at 12:00 AM, Tom S <tom-sourceforge@anonymised.com>
wrote:

So I have several questions.

Is the size limit on the Quota page supposed to reflect the layer cache
file AND the DB files, or just the former?

Just the tile cache, which is an estimate based on the block size of your
file system.

Would you normally expect a multi-GB DB file (main and index)?

Never checked honestly, the cases I've seen normally allow for hundreds of
GB for the tiles, so a multi-GB DB file normally goes un-noticed.

Is anyone successfully using Disk Quotas with Windows 2008 or later?

Haven't tried personally, would be interesting though.

If so, are the layers 'dynamic' - i.e. the data changes frequently, and
layer timeouts are specified. Could this be contributing to the issue?

Is there a way to 'dump' the quota DB files. I have not used H2, and a
quick attempt using its 'Console' does not let me open the DB file.

This is because the db file is kept locked by GeoServer, you should first
stop GeoServer.

Does anyone have any suggestions for other ways to approach this
(external DB, ...)?

Yep, using an external db should help at least seeing what's going on there.
If a delete fails during truncantion you should see something like this in
the logs:
During truncate request: <error message>

Are there any Java or logging settings I should change to help diagnose
this?

I checked a bit, does not look like there is much logging besides error
conditions, and those are logged at ERROR or WARN level already.
Basically they should show up always.

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

5

Is anyone successfully using Disk Quotas with Windows 2008 or later?

Yes I am - have a 25gb cache on windows server 2008

It is a bit fussy about reporting its size though, I am currently using 19gb but it says im using 0 bytes

Mark Ismail
Principal IT Officer
Tel: 0151 443 3021
Knowsley Council K Westmorland Road Huyton L36 9GL

Save time - use the IT Service Desk Self Serve Application

-----Original Message-----
From: Tom S [mailto:tom-sourceforge@anonymised.com]
Sent: 20 October 2014 23:09
To: geoserver-users@lists.sourceforge.net
Subject: [Geoserver-users] Integrated GeoWebCache and DiskQuota on Windows

We are attempting to implement caching and are seeing problems with the disk use. The environments we have tested are

GeoServer 2.6, 2.5.x, 2.2.x
Windows 2012 R2, Win 2008 R2
Java 7, Java 6
Tomcat 8, Tomcat 6

The environment does not make any difference - beyond the fact that this is a typical Windows deployment. It fails under all of them.

For cleanup we specified LRU (Least Recently Used). The layers we are caching are dynamic - the data updates each minute (for parts of it).
We have both server and client timeouts set on the layer. On the client end, the layers work as expected, and we also see proper behavior for the cache hits (we think). What we don't see is proper cleanup of the cache on the filesystem. There is one oddity, and one major problem.
The problem is that the specified size limits are not being honored.
For example, I have set the limit to 20GB, but the cache directory ended up with over 100GB of files before I manually cleaned it. The oddity is the size of the H2 DB files. The same system had DB files over 10GB in size. There were no relevant errors in the geoServer logs (set at Production Level logging) during the time it approached and exceeded the limit. The check frequency is set to 5 minutes, but this progression lasted for days. The DiskQuota status size used does NOT match the actual space used on disk (or even close after a while), and also rises above and stays above the set limit.

My first thought is that the system is attempting to delete the old files, but is silently failing. I have made a number of changes to the directory permissions to try to insure that is not a trigger without any improvement.

So I have several questions.

Is the size limit on the Quota page supposed to reflect the layer cache file AND the DB files, or just the former?

Would you normally expect a multi-GB DB file (main and index)?

Is anyone successfully using Disk Quotas with Windows 2008 or later?

If so, are the layers 'dynamic' - i.e. the data changes frequently, and layer timeouts are specified. Could this be contributing to the issue?

Is there a way to 'dump' the quota DB files. I have not used H2, and a quick attempt using its 'Console' does not let me open the DB file.

Does anyone have any suggestions for other ways to approach this (external DB, ...)?

Are there any Java or logging settings I should change to help diagnose this?

Any help would be appreciated.

------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users
This e-mail and any attachments are confidential. It may contain privileged information and is intended for the named recipient(s) only. It must not be distributed without consent. If you are not one of the intended recipients, please notify the sender immediately and do not disclose, distribute, or retain this email or any part of it and do not take any action based on it.

Unless expressly stated, opinions in this email are those of the individual sender, and not of Knowsley MBC. Legally binding obligations can only be created for, or be entered into on behalf of, Knowsley MBC by duly authorised officers or representatives.

Knowsley MBC excludes any liability whatsoever for any offence caused, any direct or consequential loss arising from the use, or reliance on, this e-mail or its contents. We believe but do not warrant that this e-mail and any attachments are virus free. You must therefore take full responsibility for virus checking and no responsibility is accepted for loss or damage arising from viruses or changes made to this message after it was sent. Knowsley MBC reserves the right to monitor and/or record all e-mail communications through its network in accordance with relevant legislation.

6

Hi,

We do not use disk quota but I have a some fuzzy evidence that truncating does not always remove tiles from the cache but that's on Linux. I have tried to clear old tiles from certain areas with truncate requests but some old tiles just stay there until I write new ones over them with reseed. I can imagine that it truncate fails it would lead to the trouble you have with DiskQuota.

I could use some time for debugging this but I have no idea about how to do it. Truncate does not give a list of files which it is going to delete from the cache and I haven't yet found any log data to tell if truncate processes have been successful or not. Listing files before truncate - after truncate for making a comparison is not really possible either because the cache size is about 2 TB.

-Jukka Rahkonen-
________________________________________
Ismail, Mark wrote:

Is anyone successfully using Disk Quotas with Windows 2008 or later?

Yes I am - have a 25gb cache on windows server 2008

It is a bit fussy about reporting its size though, I am currently using 19gb but it says im using 0 bytes

Mark Ismail
Principal IT Officer
Tel: 0151 443 3021
Knowsley Council K Westmorland Road Huyton L36 9GL

Save time - use the IT Service Desk Self Serve Application

-----Original Message-----
From: Tom S [mailto:tom-sourceforge@anonymised.com]
Sent: 20 October 2014 23:09
To: geoserver-users@lists.sourceforge.net
Subject: [Geoserver-users] Integrated GeoWebCache and DiskQuota on Windows

We are attempting to implement caching and are seeing problems with the disk use. The environments we have tested are

GeoServer 2.6, 2.5.x, 2.2.x
Windows 2012 R2, Win 2008 R2
Java 7, Java 6
Tomcat 8, Tomcat 6

The environment does not make any difference - beyond the fact that this is a typical Windows deployment. It fails under all of them.

For cleanup we specified LRU (Least Recently Used). The layers we are caching are dynamic - the data updates each minute (for parts of it).
We have both server and client timeouts set on the layer. On the client end, the layers work as expected, and we also see proper behavior for the cache hits (we think). What we don't see is proper cleanup of the cache on the filesystem. There is one oddity, and one major problem.
The problem is that the specified size limits are not being honored.
For example, I have set the limit to 20GB, but the cache directory ended up with over 100GB of files before I manually cleaned it. The oddity is the size of the H2 DB files. The same system had DB files over 10GB in size. There were no relevant errors in the geoServer logs (set at Production Level logging) during the time it approached and exceeded the limit. The check frequency is set to 5 minutes, but this progression lasted for days. The DiskQuota status size used does NOT match the actual space used on disk (or even close after a while), and also rises above and stays above the set limit.

My first thought is that the system is attempting to delete the old files, but is silently failing. I have made a number of changes to the directory permissions to try to insure that is not a trigger without any improvement.

So I have several questions.

Is the size limit on the Quota page supposed to reflect the layer cache file AND the DB files, or just the former?

Would you normally expect a multi-GB DB file (main and index)?

Is anyone successfully using Disk Quotas with Windows 2008 or later?

If so, are the layers 'dynamic' - i.e. the data changes frequently, and layer timeouts are specified. Could this be contributing to the issue?

Is there a way to 'dump' the quota DB files. I have not used H2, and a quick attempt using its 'Console' does not let me open the DB file.

Does anyone have any suggestions for other ways to approach this (external DB, ...)?

Are there any Java or logging settings I should change to help diagnose this?

Any help would be appreciated.

------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users
This e-mail and any attachments are confidential. It may contain privileged information and is intended for the named recipient(s) only. It must not be distributed without consent. If you are not one of the intended recipients, please notify the sender immediately and do not disclose, distribute, or retain this email or any part of it and do not take any action based on it.

Unless expressly stated, opinions in this email are those of the individual sender, and not of Knowsley MBC. Legally binding obligations can only be created for, or be entered into on behalf of, Knowsley MBC by duly authorised officers or representatives.

Knowsley MBC excludes any liability whatsoever for any offence caused, any direct or consequential loss arising from the use, or reliance on, this e-mail or its contents. We believe but do not warrant that this e-mail and any attachments are virus free. You must therefore take full responsibility for virus checking and no responsibility is accepted for loss or damage arising from viruses or changes made to this message after it was sent. Knowsley MBC reserves the right to monitor and/or record all e-mail communications through its network in accordance with relevant legislation.

------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users