Or, if you don’t want/can not to spend time yourself, you can look into commercial support and pay for someone to
build a test setup to reproduce, and eventually fix it: http://geoserver.org/comm/
···
On Mon, Feb 12, 2018 at 12:25 PM, Darell van der Voort <darellvdv@anonymised.com> wrote:
Hello everyone,
An update on the issue. I am now experiencing the problem regularly, even when adding a single account. The roles.xml keeps getting corrupted, mostly the associations between role_id and username are lost. It only happens when I access the roles.xml file, but it does seem random as to when it gets corrupted or not. Therefore it is difficult for me to reproduce the exact error. These are the steps I take:
- GET roles.xml from: /geoserver/geofence/rest/roles
- POST rolename to: /geoserver/geofence/rest/roles/role/ + rolename
- POST association to: /geoserver/geofence/rest/roles/role/ + rolename + /user/ + username
Geoserver log:
2018-02-12 09:38:00,268 INFO [security.xml] - Successful lock: security/role/default/roles.xml.lock
2018-02-12 09:38:00,268 INFO [geoserver.security] - Start storing roles for service named default
2018-02-12 09:38:00,389 INFO [geoserver.security] - Storing roles successful for service named default
2018-02-12 09:38:00,389 INFO [geoserver.security] - Start reloading roles for service named default
2018-02-12 09:38:02,842 INFO [geoserver.security] - Reloading roles successful for service named default
2018-02-12 09:38:02,842 INFO [geoserver.security] - Adjusted last modified for file: security/role/default/roles.xml
2018-02-12 09:38:03,520 INFO [security.xml] - Successful lock: security/role/default/roles.xml.lock
2018-02-12 09:38:03,520 INFO [geoserver.security] - Start storing roles for service named default
2018-02-12 09:38:03,638 INFO [geoserver.security] - Storing roles successful for service named default
2018-02-12 09:38:03,639 INFO [geoserver.security] - Start reloading roles for service named default
2018-02-12 09:38:05,886 INFO [geoserver.security] - Reloading roles successful for service named default
2018-02-12 09:38:05,887 INFO [geoserver.security] - Adjusted last modified for file: security/role/default/roles.xml
2018-02-12 09:38:06,157 INFO [geoserver.security] - Start reloading roles for service named default
2018-02-12 09:38:08,526 INFO [geoserver.security] - Reloading roles successful for service named default
2018-02-12 09:38:08,529 INFO [geoserver.security] - Adjusted last modified for file: security/role/default/roles.xml
2018-02-12 09:38:08,534 INFO [geoserver.security] - Start reloading user/groups for service named default
2018-02-12 09:38:09,613 INFO [geoserver.security] - Reloading user/groups successful for service named default
2018-02-12 09:38:09,613 INFO [geoserver.security] - Adjusted last modified for file: security/usergroup/default/users.xml
Looking at the log it leaves to believe it has something to do with the .lock file. As our database keeps getting bigger, the problem arises more often. I believe that writing the roles.xml takes a longer time if there are more users/roles. The two POST requests are executed directly after each other, therefore trying to access the roles.xml file before it is completely written by the first POST request. Shouldn’t the lock file prohibit this kind of behavior? I am working on a solution that checks for the .lock file before accessing and writing to roles.xml as I believe this is the source of the error.
Kind regards,
Darell
–
On Wed, Jan 24, 2018 at 10:01 AM, Darell van der Voort <darellvdv@anonymised.com> wrote:
Hello,
Thank you for the thorough explanation Harleen, very helpful. It turned out that also our roles.xml was corrupted in production. It did keep most of the role_ids, however it removed nearly all of the role-user associations, including the admin. Hence the lock out. We ware able to restore most of the accounts by re-adding the associations. Also the Geofence data rules were missing for most of the workspaces/accounts.
I believe the corrupt file occurs when doing to much I/O to the xml files. The first error before the 403 was a xml.etree.ElementTree.ParseError. After that, the roles.xml was mostly lost. I am able to reproduce the error on our staging environment by making a lot of API calls to add users and check the xml for existing users. A solution that worked for me is leaving enough time between the calls. Last night I’ve added 450+ users, roles and workspaces without any problems. There were however a few minutes between each user creation as there was a data processing and uploading step in between. I am now backing up the users.xml and roles.xml regularly, losing those account associations in production had a huge impact.
Andrea, I hope to find some time to make a vanilla installation with some descriptions and the script.
Thanks again for the great help!
Kind regards,
Darell
On Wed, Jan 24, 2018 at 9:26 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:
Hi all,
thanks for moving the discussion forward. Just to be clear, the only way a developer may look into this
is that you make it easily reproducable on a vanilla installation that anyone can run on their own.
If GeoFence has to be plugged in to reproduce, that is not a problem, just describe it, and provide
that and the script to make it happen in a ticket at https://osgeo-org.atlassian.net/projects/GEOS/summary
Cheers
Andrea
On Tue, Jan 23, 2018 at 9:25 PM, Harleen Lappano <hlappano@anonymised.com> wrote:
Hey again,
Just to add I hit the snag again after making those above changes too. However, I isolated the situation more and how to sort of recover back.
The roles.xml file is getting corrupted somehow. It is removing some of the users that were already in place including admin. Which is why the forbidden 403 status is occurring. By syncing up the roles.xml file again with admin, you will be able to get access again. But you do need the interval check tag set in the config in order for geoserver to read in the external changes. Or I think you can restart geoserver, not sure about that though.
However, just don’t know what is causing it to get corrupted. The users are getting added correctly to the users.xml file, but the roles on the other hand is not becoming off sync.
–
Regards,
Andrea Aime
==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.
The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.
On Tue, Jan 23, 2018 at 10:47 AM, Harleen Lappano <hlappano@anonymised.com> wrote:
Hey,
Andrea, I also don’t know if we will be able to run a test here without Geofence. It would be on my own time right now since we battled through the issue for now. However, I can say we tried to dump a users.xml and roles.xml while it was running to add the users that way instead of the REST api and that did however cause an issue when the Authentication Cache went to run. We also attempted to add one user that way too. Same issue. I’m not sure if we just hit it with luck on that one user, but it did have the same effect.
Darell as far as the steps go I will list what we do. Since we are just pushing GeoFence up for the first time to production our redeploy envolves a bit and it uses ansible. However, I’ll try to list it hope it makes sense.
Also by doing this the existing users will be lost, so make sure you have a backup or a way to get them back in.
We currently have two user group services. The default one to keep admin credentials on. And a second one for our security authentication. It is the second one that holds all the users.
Steps we took:
- Stop Tomcat Service
- Install GeoFence
- Unarchive geoserver-{{geofence_version}}-SNAPSHOT-geofence-server-plugin.zip (This is the embedded version)
- Drop them in the tomcat/webapps/geoserver/WEB-INF/lib/
- Change permissions, owner and group. Which ever one is set for your tomcat. Permissions 06441. Copy Security Files (These are the ones we made modifications for our setup).
-
List of files Under /tomcat/webapps/geoserver/data/security/
-
config.xml (This contained our filter chains)
-
rest.properties (we made a change here for access on a specific role)
-
/filter/{{our_authentication}}/config.xml
-
/role/default/
-
/config.xml (this has the change I mentioned, however, we changed it after deploy because we were using our deploy package)
-
/roles.xml
-
(the roles.xsd gets generated after first start up)- /usergroup/{{our_usergroup_name}}/ (this is a new one we created that is not default.)
-
/config.xml
-
/users.xml
-
/users.xsd1. Start Tomcat Service
-
Wait For Tomcat to Start
-
Add Data Rules
-
These are our data rules using Geofence REST.1. After this we go to the admin page and see we have control again.
-
We also ran a test with our admin user, so this might have triggered the authentication cache. Mabe best to wait 10 minutes till your cache refreshes before adding the users.
-
Stop Tomcat Service
-
Modify users.xml and roles.xml as mentioned in the last email.
-
Start Tomcat Service
-
Ran our Users script
-
Loop through all users
-
curl REST API to add user
-
curl REST API to associate user with role
-
sleep 1 second (Not sure if we need this. Just noticed without it, it will chug and possibly run into a stack overflow exception)
And those my friend our the steps we took. Like I said we use ansible, so we never isolated what exactly caused it to unlock. Actually listing this out, I am wondering if it was the refresh of users.xml and roles.xml to a blank state.
Hope that helps.
–
Regards,
Harleen Lappano
407-473-7157
On Tue, Jan 23, 2018 at 6:45 AM, Darell van der Voort <darellvdv@anonymised.com> wrote:
Hello Harleen, Andrea,
Thank you very much for the input! Harleen, your problem is exactly the same as we are facing. I am also adding authentication with geofence and pull the exisiting users from the users.xml file. Normally we only add just a few users without any problems. But indeed with more than 200 users the lockdown happens.
I have just pushed a new Geoserver build to production and changed both config files to checkInterval 0 and validating to false. However the problem still persists. I have rebooted the instances and restarted all the task definitions. What do you mean exactly with ‘redeploy Geofence and security settings including the users.xml and roles.xml’? We have those settings on a shared mount so that each Geoserver instance uses the same settings. What steps did you do to redeploy, without creating new user and role xmls?
Andrea, I do not thing it can be reproduced without Geofence in the mix, as this is the only way to add users via the REST API. I can however reproduce it with Geofence. Also thanks for the support link.
Kind regards,
Darell
–
Regards,
Harleen Lappano
407-473-7157
On Tue, Jan 23, 2018 at 10:47 AM, Andrea Aime <andrea.aime@anonymised.com> wrote:
Hi Darell and Harleen,
it seems you’re on to something. Wondering, can it be reproduced without GeoFence in the mix?
It would be useful to have a script that reproduces the problem on a vanilla GeoServer installation, if that can be done
please open a ticket with it at: https://osgeo-org.atlassian.net/projects/GEOS/summary
Darell, this list offers volountary support, which does not do “urgent” (well, not with guarantees at least, it can happen of course).
If you’re in a hurry I’d suggest you also check http://geoserver.org/support/
Cheers
Andrea
On Tue, Jan 23, 2018 at 1:50 AM, Harleen Lappano <hlappano@anonymised.com> wrote:
Hey Darell,
I’m not a expert on Geoserver. However, funny story is that I ran into this same issue last Friday as we were pushing our changes to our production.
We are using Geoserver 2.9.1 for our product. I don’t remember the tomcat version.
If it wasn’t for your email, we would have been still trying to trace down all our changes and deltas we made for this push window, so I appreciate that you posted this it helped us narrow down and push pass this issue.
We were adding authentication to our geoserver with geofence so we can add users using the REST api. I did a similar script where I pulled our existing users in our system to also add them to the geoserver’s user list using REST api. At some point in our process we had a total lock down on our geoserver in production. So just to see I ran a script on our dev box to add 200 random users to the geoserver and was able to reproduce what happened in our production.
This is what we did based on our observations of the problem and it seemed to get pass this issue for now. We are still running tests this week on our product, however, mabe it will help you get along as well.
We are thinking that there might be an issue in geoserver regarding the users.xml or roles.xml. It seems like the 3 components that might be the cause are 1. Authentication Cache that runs every 10 minutes, 2. Our curl commands in our script that adds the user. and 3. The interval check for modifications on the users.xml or roles.xml. (these are the names we chose, but it is the xml that holds the user group service and the role server).
In order for us to get back to an unlock state, I had to redeploy our GeoFence and security settings including the users.xml and roles.xml. I’m not sure what is in the set that causes the geoserver to unlock because we didn’t trace it down that far.
Then we went ahead and changed our config.xml for our user group and roles has (two config files):
0
false
We weren’t sure about validating, but we turned it off anyways. After that and a restart, we were able to run our script to add the users without any problems. I will add that in our script we added a second delay after each add. I noticed in the original time we did our script we ran into a stack overflow issue.
After all this I think we are concluding that there might have been a lock on the file resource when reading and writing. However, this is just a hunch without further testing.
Hopefully someone on this mailing list knows exactly what is happening.
Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
Geoserver-users mailing list
Please make sure you read the following two resources before posting to this list:
If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer
Geoserver-users@anonymised.comrge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users
–
Regards,
Andrea Aime
==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.
The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.
On Mon, Jan 22, 2018 at 5:02 AM, Darell van der Voort <darellvdv@anonymised.com> wrote:
Hello everybody,
I’m running Geoserver 2.11.2 on Java 1.8.0_144 32 bits on Amazon Linux together with Tomcat 9. I have a very urgent problem.
I have a Python script that uses the REST API to automatically create users, set read/write permissions and create workspaces. I have been using this for the last months without any problems. However today I had to create 450+ accounts and workspace so I wrote a loop to generate the accounts. This went flawless till about account 137. After that I got a access denied:
HTTP 403 status Forbidden
Access is denied
The server understood the request but refuses to authorize it.
Apache Tomcat/9.01
Also when I log into the web interface using the admin account I cannot access anything. The data is however still served. This leads me to believe it is some kind of protection from Geoserver against DDoS or brute force attacks. I do not have another account with admin rights.
How do I re-enable access to the Geoserver? I already tried restarting the server instance, but no luck.
Thanks!
Darell
Check out the vibrant tech community on one of the world’s most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
Geoserver-users mailing list
Please make sure you read the following two resources before posting to this list:
If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer
Geoserver-users@anonymised.comrge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users
–
Regards,
Harleen
Regards,
Andrea Aime
==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.
The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.