Hi group,
I’ve just created this PR <https://github.com/geoserver/geoserver/pull/3450>, which speeds up GeoServer initialization when there are lots of datastore info objects to load.
I’ve a data directory with about 40K shapefile datastores (no layers, in order to check this particular patch) and startup time on my machine goes from 38s to 13s.
PR review most appreciated.
Cheers,
Gabriel Roldán
More on startup time, added the corresponding layers to those 40k shapefile stores, and geoserver was choking in loading the gwc tile layers from datadir/gwc-layers.
The following PR[1] takes loading the tile layers back down to ~1.5s instead of ~183s. With #3450, full start up down from 280s to 24s.
It is too late now, tomorrow/Monday I’ll try to create a jira issue to the whole startup time thing.
Again, a PR review would be appreciated.
Cheers,
Gabriel.
[1] <https://github.com/geoserver/geoserver/pull/3451>
Gabriel Roldán
Hi Gabriel,
I also noticed this one, thanks a lot for fixing it, it really got me mad when I saw it months ago but was
really hoping someone else to notice it and do something about it (I’m really sick of having to fix after
a certain developer mess).
Cheers
Andrea
Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.
Thank you for that Gabriel, I have immediately reviewed your PR since I am responsible. I have been mostly working with jdbc configurations which is why I was not aware of causing this performance issue in a file-based setup.
I would like to call out again to all geoserver developers that I am always available to do for reviews in a reasonable time frame, especially if it is code that I have written or am very familiar with. So please do not hesitate to assign me if github suggests me.
Also, if a regression of this kind becomes apparent only after review and merge, obviously report it asap, and you can definitely bring it to my attention too. I have never responded in any other way then to make it a priority to fix things I did wrong. I have always done my best to contribute in the best way possible, including fixing many bugs and adding test coverage to prevent regression, but mistakes do happen with anyone. I enjoy cooperating in a positive way with anyone willing to do so, like I have recently been doing with Steve on jdbcconfig. My only shortcoming in this community is that I don’t always manage to follow everything that is going on, but if you contact me directly I will always respond very rapidly and do everything to assist.
Kind Regards
Niels
On 28/04/2019 08:29, Gabriel Roldan wrote:
More on startup time, added the corresponding layers to those 40k shapefile stores, and geoserver was choking in loading the gwc tile layers from datadir/gwc-layers.
The following PR[1] takes loading the tile layers back down to ~1.5s instead of ~183s. With #3450, full start up down from 280s to 24s.
It is too late now, tomorrow/Monday I’ll try to create a jira issue to the whole startup time thing.
Again, a PR review would be appreciated.
Cheers,
Gabriel.[1] <https://github.com/geoserver/geoserver/pull/3451>
El sáb., 27 de abr. de 2019 a la(s) 20:59, Gabriel Roldan (gabriel.roldan@anonymised.com) escribió:
Hi group,
I’ve just created this PR <https://github.com/geoserver/geoserver/pull/3450>, which speeds up GeoServer initialization when there are lots of datastore info objects to load.
I’ve a data directory with about 40K shapefile datastores (no layers, in order to check this particular patch) and startup time on my machine goes from 38s to 13s.
PR review most appreciated.
Cheers,
Gabriel Roldán
–
Gabriel Roldán
_______________________________________________ Geoserver-devel mailing list [Geoserver-devel@lists.sourceforge.net](mailto:Geoserver-devel@lists.sourceforge.net) [https://lists.sourceforge.net/lists/listinfo/geoserver-devel](https://lists.sourceforge.net/lists/listinfo/geoserver-devel)
Thank you for that Gabriel, I have immediately reviewed your PR since I am responsible. I have been mostly working with jdbc configurations which is why I was not aware of causing this performance issue in a file-based setup.
This is problematic, as file based setup is the core functionality and jdbc configuraton is an unsupported community module (I’ll notice that questions about jdbconfig still remain un-answered on the user list, despite your claim here on geoserver-devel to be the “de-facto maintainer” of the module).
Whatever change you do in core, it must be tested first and foremost against the normal functionality with ship with, regardless of your opinion on it.
I would like to call out again to all geoserver developers that I am always available to do for reviews in a reasonable time frame, especially if it is code that I have written or am very familiar with. So please do not hesitate to assign me if github suggests me.
Also, if a regression of this kind becomes apparent only after review and merge, obviously report it asap, and you can definitely bring it to my attention too. I have never responded in any other way then to make it a priority to fix things I did wrong. I have always done my best to contribute in the best way possible, including fixing many bugs and adding test coverage to prevent regression, but mistakes do happen with anyone. I enjoy cooperating in a positive way with anyone willing to do so, like I have recently been doing with Steve on jdbcconfig. My only shortcoming in this community is that I don’t always manage to follow everything that is going on, but if you contact me directly I will always respond very rapidly and do everything to assist.
As long as your activity is confined to unsupported modules that is fine, but it becomes a problem when you perform changes
against supported modules, core devs should not be forced to chase after you. If you cannot be active on lists (both devel and user),
it’s best if you refrain from proposing changes in core or supported extensions, other than simple (small, well contained) bug fixes.
The reason is simple, if you don’t follow them up with active support, those changes make the life of core devs harder:
but this is also nasty, as this places an extra burden on PR reviews, which are most of the time an unpaid activity.
This wiki page back in 2016 is also relevant to this discussion (your work often falls in the “Other contributors providing large changes” section):
https://github.com/geoserver/geoserver/wiki/Some-ideas-about-minimum-community-software-responsibility-and-participation
The GeoServer PR list currently has 13 open pull requests of yours, some have been sitting there for months.
Am I right in assuming that those are all related to work that you’ve been paid to bring to GeoServer?
While I take my volunteered status of maintainer seriously, I don’t see why I should be using my unpaid spare time on them, with an outlook of potentially having to spend more unpaid time
answering user questions about their functionality, and eventually fixing bugs and handling regressions that they might have caused.
Given the number of PRs, it’s also going to be a very significant amount of time.
It may be that others feel the same, as none of the PRs received, despite months of being out
(after 3 months out there, I guess we can rule out a “lack of time” to review and default on a more active decision not to look at them).
A quick statistics using:
git shortlog -nsu --since “three years ago”
shows you’re one of the top 10 commiters in GeoServer, with 102 commits (they need to be summed up from two different nicks):
1342 Andrea Aime
510 Torben Barsballe
184 Nuno Oliveira
171 Jody Garnett
128 Alessio Fabiani
128 Ben Caradoc-Davies
104 Daniele Romagnoli
99 Kevin Smith
93 afabiani
91 Erik Merkle
79 Steve Ikeoka
77 Brad Hards
77 Ian Turton
67 NielsCharlier
56 Mike Pumphrey
36 Simone Giannecchini
35 Mauro Bartolomeoli
35 Niels Charlier
The other developers contributing a similar amount of commits are actually active in the community with a number
of unpaid activities that help the project stay alive, on the user lists answering questions, on the dev list participating
to discussion, participating to the monthly bug fix code sprint, and cutting releases.
Jody once quite correctly called it “the cost of doing business in a open source project”.
Given that your activity is not a charity, but it’s instead paid for, business, it’s only fair to expect the same of you.
Regards,
Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.
That is fair enough, but it regards a reduction in start-up performance that only becomes apparent if you have a large amount of cached layers in your setup, it will not necessarily be detected with just a basic testing environment. There is no mechanism in place to detect regressions of this sort, and neither me nor the reviewer detected it. I’m not the first and I won’t be the last who makes a mistake like that. Something like that is actually harder to prevent than breaking a build, which occasionally happens to people too. We all have the ability to miss things. I don’t see the point in getting mad about it, as long as people take their responsibility, just communicate.
On 29/04/2019 09:06, Andrea Aime wrote:
This is problematic, as file based setup is the core functionality and jdbc configuraton is an unsupported community module (I’ll notice that questions about jdbconfig still remain un-answered on the user list, despite your claim here on geoserver-devel to be the “de-facto maintainer” of the module).
Whatever change you do in core, it must be tested first and foremost against the normal functionality with ship with, regardless of your opinion on it.
What is far more bizarre is that some gets really mad about, but decides to not communicate or report it for months and then launch passive aggressive personal attacks when someone else reports (and fixes) it.
As long as your activity is confined to unsupported modules that is fine, but it becomes a problem when you perform changes
against supported modules, core devs should not be forced to chase after you. If you cannot be active on lists (both devel and user),
it’s best if you refrain from proposing changes in core or supported extensions, other than simple (small, well contained) bug fixes.
Unlike what you seem to suggest, I am pretty sure that all or most of what I have done in the past few years that is not in community modules, are actually just bug fixes. (I guess it is disputable whether making config pages cluster safe is a feature or a bug.) But any major core development has been a long time.
But yes, you are right, my presence on the mailing lists is often lacking because of being busy, and I do sometimes miss emails for which I am the ideal person to respond to. Although I am someone who always responds very quickly when emails are directed to me (or I get asked to review) , it should not be anyone’s job to CC or forward to me. I am at fault there, and I have admitted this before (with improvements, unfortunately only temporarily). You can definitely complaint and call me up on this, but whether your attitude towards me is proportional to this is another matter.
I have done and still do in fact contribute with volunteering work, such as reviews (including for colleagues of yours) and maintenance, including for code I wrote for a company who paid me to develop certain modules or core changes but never intended to pay me to maintain these. I can always be contacted about this work, and I have taken that personal responsibility upon myself entirely, and I am always willing to continue to do so. As far as that certain company is considered, they couldn’t care less if I had developed that stuff and disappeared afterwards, not even being contactable about it.
The other developers contributing a similar amount of commits are actually active in the community with a number
of unpaid activities that help the project stay alive, on the user lists answering questions, on the dev list participating
to discussion, participating to the monthly bug fix code sprint, and cutting releases.
Jody once quite correctly called it “the cost of doing business in a open source project”.Given that your activity is not a charity, but it’s instead paid for, business, it’s only fair to expect the same of you.
Not all major contributors are active any more. I cannot really commit to participate in meetings and releases, because I cannot commit to having enough availability at the right time unfortunately to do this as a volunteer because of other obligations. Responding to emails and making reviews as a volunteer is something else entirely, because I can do that at any time, in the middle of the night if need be. After all, my own personal time is the only resource I have for this, and I do the best I can to help the geoserver community. I am not a profit-making business who sells products and services based on geoserver, I am just a very simple freelance programmer on his own. I don’t know if there are many others who have participated an entire week of code sprint purely as a personal volunteer, paying everything except for the flight out of their own pocket, and missing a week’s worth of work in opportunity cost - not in the context of a larger company.
The reasons that I still contribute quite a bit to geoserver at this moment is because
I do these things to do good, not to do evil. I believe in this model of development, and I also wished companies would invest more in the community work. Again, I am not a business, just a simple programmer, and I could easily earn the same living without caring at all about geoserver and its community. This picture you want to paint of me as a leech who takes and profiteers but never cares or gives is just not close to reality. I simply do my best with the resources I have, but I am not perfect.
The GeoServer PR list currently has 13 open pull requests of yours, some have been sitting there for months.
Am I right in assuming that those are all related to work that you’ve been paid to bring to GeoServer?
While I take my volunteered status of maintainer seriously, I don’t see why I should be using my unpaid spare time on them, with an outlook of potentially having to spend more unpaid time
answering user questions about their functionality, and eventually fixing bugs and handling regressions that they might have caused.
Given the number of PRs, it’s also going to be a very significant amount of time.
It may be that others feel the same, as none of the PRs received, despite months of being out
(after 3 months out there, I guess we can rule out a “lack of time” to review and default on a more active decision not to look at them).
There have been several people willing to review PR’s of mine the past few months, the only ones that seem to remain ignored are the CSW ones, and I assume this is because all developers familiar with this module are from your company.
The vast majority of these are small bug fixes (a few exceptions - some small improvements and one big one; all well documented and unit tested).
There is not much need for additional support, because this concerns functionality that geoserver according to the documentation already provides: the iso metadata profile, but which currently really doesn’t work properly. The alternative is to have a new CSW community module with iso metadata profile properly supported. I do think if even the small bug fixes get ignored, this is a personal rather than pragmatic matter. I don’t see how any of the CSW stuff I did can cause problems for anyone.
I don’t expect people to do all this for me out of the goodness of their heart, I was in all honesty hoping that people would be able to use this. Admittedly, I also like to avoid that maintaining forks becomes to much of a complicated matter. I understand the bigger reviews are more tricky, but I am very willing to trade time for this and for example, get assigned some reviews and/or bug fixes in exchange.
Regards
Niels
What is far more bizarre is that some gets really mad about, but decides to not communicate or report it for months and then launch passive aggressive personal attacks when someone else reports (and fixes) it.
I have lost any faith that you will fix whatever regressions you cause, that’s why I did not communicate, I was sure you’d have put up excuses and not do anything, which would have just made things worse.
Just recently we had a discussion on gwc-devel about a randomly failing test that you introduced, I believe your comment was that you just adapted a test that Gabriel made. No fix was provided on your part, the random failure is still there.
Unlike what you seem to suggest, I am pretty sure that all or most of what I have done in the past few years that is not in community modules, are actually just bug fixes. (I guess it is disputable whether making config pages cluster safe is a feature or a bug.) But any major core development has been a long time.
Yes, the large works are a few years old (app-schema, Resource, CSW ISO), but I saw no indication that anything changed in your attitude towards the project and maintenance of your work. Hence my assumption above.
I have done and still do in fact contribute with volunteering work, such as reviews (including for colleagues of yours) and maintenance, including for code I wrote for a company who paid me to develop certain modules or core changes but never intended to pay me to maintain these. I can always be contacted about this work, and I have taken that personal responsibility upon myself entirely, and I am always willing to continue to do so. As far as that certain company is considered, they couldn’t care less if I had developed that stuff and disappeared afterwards, not even being contactable about it.
This is problematic, because if you don’t maintain your contributions to supported modules, someone else will have to, and that someone is a handful of already heavily overloaded core developers, often using significant amounts of their spare time every single week. I think GeoServer as a project has no capacity left to receive “contribution dumps” where the developers provides code and then leaves to do something else.
The other developers contributing a similar amount of commits are actually active in the community with a number
of unpaid activities that help the project stay alive, on the user lists answering questions, on the dev list participating
to discussion, participating to the monthly bug fix code sprint, and cutting releases.
Jody once quite correctly called it “the cost of doing business in a open source project”.Given that your activity is not a charity, but it’s instead paid for, business, it’s only fair to expect the same of you.
Not all major contributors are active any more.
Yes, that is indeed part of the above problem, the ones that are left are more and more overworked, not only from the point of view of the amount of actual coding to do, but more importantly in terms of cognitive load, one thing having to manage 2-5 modules, another entirely is to have to manage and be proficient with several tens of them, I cannot comment about others, but personally I am working beyond my limits.
I cannot really commit to participate in meetings and releases, because I cannot commit to having enough availability at the right time unfortunately to do this as a volunteer because of other obligations. Responding to emails and making reviews as a volunteer is something else entirely, because I can do that at any time, in the middle of the night if need be. After all, my own personal time is the only resource I have for this, and I do the best I can to help the geoserver community. I am not a profit-making business who sells products and services based on geoserver, I am just a very simple freelance programmer on his own. I don’t know if there are many others who have participated an entire week of code sprint purely as a personal volunteer, paying everything except for the flight out of their own pocket, and missing a week’s worth of work in opportunity cost - not in the context of a larger company.
That participation was appreciated, but what keeps the project alive is day to day participation. Think of it as having to keep up with a cow farm, you can show up and do something wonderful for a week long, but if you then leave them for a year and nobody cares about them, by you come back all the cows are dead.
I do these things to do good, not to do evil. I believe in this model of development, and I also wished companies would invest more in the community work. Again, I am not a business, just a simple programmer, and I could easily earn the same living without caring at all about geoserver and its community. This picture you want to paint of me as a leech who takes and profiteers but never cares or gives is just not close to reality. I simply do my best with the resources I have, but I am not perfect.
You are a business, you just happen to be a single person business, but what you do is similar to what GeoSolutions does, we don’t sell a downstream product, simply provide improvement and bug fixing services (so in the end we get paid by the hours worked). And then we dedicate part of the company time, and in the case of some of us, part of personal time, to keep that code going, once the contract is long gone.
Your portion of business relating to GeoServer exists because someone keeps the project alive by answering user questions, doing releases, participating to the monthly bug fix sprints and the like, without being paid to do that.
I understand that you don’t have the same amount of activity and don’t pretend you show up for releases once every two months like we do, or be the most active dev on the user list, or participate at most bug fix code sprints, but at some of those (releases, bug fix sprints, fielding user questions), yes, you should.
There is not much need for additional support, because this concerns functionality that geoserver according to the documentation already provides: the iso metadata profile, but which currently really doesn’t work properly. The alternative is to have a new CSW community module with iso metadata profile properly supported. I do think if even the small bug fixes get ignored, this is a personal rather than pragmatic matter. I don’t see how any of the CSW stuff I did can cause problems for anyone.
It is personal, and I’ve said so multiple times in multiple places. Business is indeed not personal, it’s exchanges of money with a service, I don’t have to like the customer. Using spare time is a personal choice instead, and I do all my un-paid reviews on Saturday and Sunday mornings (for the joy of those contributing in paid hours, and wondering why it takes so much time to get one and so much un-synchronized with their needs to get code in).
This makes open source projects something about people, unlike other things like healthcare where one can expect a service in a impersonal way (but only because taxes were paid for it). So yes, personal is the norm in free software communities, you’ll have to learn to deal with it.
About ISO support, I did not work on that part of the CSW machinery (only covered dublin core and the generic service engine), in order to review the changes I’d first have to learn about what an ISO record is, and how the code producing it works. That would require even more spare time to be used.
I don’t expect people to do all this for me out of the goodness of their heart, I was in all honesty hoping that people would be able to use this.
I can see the usefulness of the work done (extension of ISO metadata support to cover more elements), I’m simply not willing to take on its review and long term maintenance myself. Again, sorry to sound like a broken record, but if a module is supported, it needs someone to willing to maintain it, adding new code just makes live harder for those that have volunteered to do such maintenance. I volunteer to maintain the code I write, not other people’s code.
Admittedly, I also like to avoid that maintaining forks becomes to much of a complicated matter. I understand the bigger reviews are more tricky, but I am very willing to trade time for this and for example, get assigned some reviews and/or bug fixes in exchange.
Hopefully someone will show up that is willing to trade horses with you here. Good luck! 
Cheers
Andrea
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.
Again, my CSW PR’s are 80% bug fixes. Trying to sabotage these for personal reasons, well that is on you.
Also, admittedly, they are bug fixes for… my own code. I originally wrote the ISO metadata profile for CSW. But it was apparently never put to use, because the schemes don’t even pas validation, or can be harvested from geonetwork, and bugs were never reported.
So in your own logic, what you’re basically saying is that you’d rather support a faulty, buggy feature that I wrote than an actual functional feature that I wrote.
About your accusations, this is what actually happened:
I did not at all introduce a failing test, I created an offline test environment, so that a whole series of test methods that were part of an online test class, could from then on actually be run with every build. This obviously makes the software more robust and helps to detect regressions. Doing this, exposed that one of these tests mysteriously fails at random. (So to be clear: the only reason that didn’t happen previously was because it was never run. The issue was exposed, not caused, by me.) I didn’t understand the test so I was hoping for some input from the author of the test, who was however not active at that time. But the discussion continued in which I was still involved as well as Kevin, and we both agreed it was better to put that single method in @ignore until the mystery was resolved, since that would still be better than the previous situation before I made the offline test environment and none of the tests were ever run.
You bend the truth to paint an entirely false picture of me, and to excuse your own strange behavior. People that know me, know that I am actually quite a bit perfectionist and code-proud; and rather keep searching for hours and hours to really understand a problem than to introduce an ugly work-around and forget about it. I have never, ever “put up excuses and not do anything” in response to an issue that I caused.
The truth is that you have acted rather hostile to me, for a very, very long time. I have never understood why. Perhaps I annoy you somehow. But nobody else in this community has ever acted this hostile to me. On the other hand, I do think other people have sometimes experienced hostility from your end. You say that things are personal in a community like this. I agree. This community has written down rules, and that is great. But people skills are also import for a community: such as being friendly and nice, not assuming the worst from people, and using constructive communication. Everybody has their own situation, their own background and circumstances, their own strengths and shortcomings, and in any sort of professional environment you have to deal with that.
Regards
Niels
On 29/04/2019 16:52, Andrea Aime wrote:
On Mon, Apr 29, 2019 at 3:36 PM Niels Charlier <niels@anonymised.com> wrote:
What is far more bizarre is that some gets really mad about, but decides to not communicate or report it for months and then launch passive aggressive personal attacks when someone else reports (and fixes) it.
I have lost any faith that you will fix whatever regressions you cause, that’s why I did not communicate, I was sure you’d have put up excuses and not do anything, which would have just made things worse.
Just recently we had a discussion on gwc-devel about a randomly failing test that you introduced, I believe your comment was that you just adapted a test that Gabriel made. No fix was provided on your part, the random failure is still there.Unlike what you seem to suggest, I am pretty sure that all or most of what I have done in the past few years that is not in community modules, are actually just bug fixes. (I guess it is disputable whether making config pages cluster safe is a feature or a bug.) But any major core development has been a long time.
Yes, the large works are a few years old (app-schema, Resource, CSW ISO), but I saw no indication that anything changed in your attitude towards the project and maintenance of your work. Hence my assumption above.
I have done and still do in fact contribute with volunteering work, such as reviews (including for colleagues of yours) and maintenance, including for code I wrote for a company who paid me to develop certain modules or core changes but never intended to pay me to maintain these. I can always be contacted about this work, and I have taken that personal responsibility upon myself entirely, and I am always willing to continue to do so. As far as that certain company is considered, they couldn’t care less if I had developed that stuff and disappeared afterwards, not even being contactable about it.
This is problematic, because if you don’t maintain your contributions to supported modules, someone else will have to, and that someone is a handful of already heavily overloaded core developers, often using significant amounts of their spare time every single week. I think GeoServer as a project has no capacity left to receive “contribution dumps” where the developers provides code and then leaves to do something else.
The other developers contributing a similar amount of commits are actually active in the community with a number
of unpaid activities that help the project stay alive, on the user lists answering questions, on the dev list participating
to discussion, participating to the monthly bug fix code sprint, and cutting releases.
Jody once quite correctly called it “the cost of doing business in a open source project”.Given that your activity is not a charity, but it’s instead paid for, business, it’s only fair to expect the same of you.
Not all major contributors are active any more.
Yes, that is indeed part of the above problem, the ones that are left are more and more overworked, not only from the point of view of the amount of actual coding to do, but more importantly in terms of cognitive load, one thing having to manage 2-5 modules, another entirely is to have to manage and be proficient with several tens of them, I cannot comment about others, but personally I am working beyond my limits.
I cannot really commit to participate in meetings and releases, because I cannot commit to having enough availability at the right time unfortunately to do this as a volunteer because of other obligations. Responding to emails and making reviews as a volunteer is something else entirely, because I can do that at any time, in the middle of the night if need be. After all, my own personal time is the only resource I have for this, and I do the best I can to help the geoserver community. I am not a profit-making business who sells products and services based on geoserver, I am just a very simple freelance programmer on his own. I don’t know if there are many others who have participated an entire week of code sprint purely as a personal volunteer, paying everything except for the flight out of their own pocket, and missing a week’s worth of work in opportunity cost - not in the context of a larger company.
That participation was appreciated, but what keeps the project alive is day to day participation. Think of it as having to keep up with a cow farm, you can show up and do something wonderful for a week long, but if you then leave them for a year and nobody cares about them, by you come back all the cows are dead.
I do these things to do good, not to do evil. I believe in this model of development, and I also wished companies would invest more in the community work. Again, I am not a business, just a simple programmer, and I could easily earn the same living without caring at all about geoserver and its community. This picture you want to paint of me as a leech who takes and profiteers but never cares or gives is just not close to reality. I simply do my best with the resources I have, but I am not perfect.
You are a business, you just happen to be a single person business, but what you do is similar to what GeoSolutions does, we don’t sell a downstream product, simply provide improvement and bug fixing services (so in the end we get paid by the hours worked). And then we dedicate part of the company time, and in the case of some of us, part of personal time, to keep that code going, once the contract is long gone.
Your portion of business relating to GeoServer exists because someone keeps the project alive by answering user questions, doing releases, participating to the monthly bug fix sprints and the like, without being paid to do that.
I understand that you don’t have the same amount of activity and don’t pretend you show up for releases once every two months like we do, or be the most active dev on the user list, or participate at most bug fix code sprints, but at some of those (releases, bug fix sprints, fielding user questions), yes, you should.There is not much need for additional support, because this concerns functionality that geoserver according to the documentation already provides: the iso metadata profile, but which currently really doesn’t work properly. The alternative is to have a new CSW community module with iso metadata profile properly supported. I do think if even the small bug fixes get ignored, this is a personal rather than pragmatic matter. I don’t see how any of the CSW stuff I did can cause problems for anyone.
It is personal, and I’ve said so multiple times in multiple places. Business is indeed not personal, it’s exchanges of money with a service, I don’t have to like the customer. Using spare time is a personal choice instead, and I do all my un-paid reviews on Saturday and Sunday mornings (for the joy of those contributing in paid hours, and wondering why it takes so much time to get one and so much un-synchronized with their needs to get code in).
This makes open source projects something about people, unlike other things like healthcare where one can expect a service in a impersonal way (but only because taxes were paid for it). So yes, personal is the norm in free software communities, you’ll have to learn to deal with it.About ISO support, I did not work on that part of the CSW machinery (only covered dublin core and the generic service engine), in order to review the changes I’d first have to learn about what an ISO record is, and how the code producing it works. That would require even more spare time to be used.
I don’t expect people to do all this for me out of the goodness of their heart, I was in all honesty hoping that people would be able to use this.
I can see the usefulness of the work done (extension of ISO metadata support to cover more elements), I’m simply not willing to take on its review and long term maintenance myself. Again, sorry to sound like a broken record, but if a module is supported, it needs someone to willing to maintain it, adding new code just makes live harder for those that have volunteered to do such maintenance. I volunteer to maintain the code I write, not other people’s code.
Admittedly, I also like to avoid that maintaining forks becomes to much of a complicated matter. I understand the bigger reviews are more tricky, but I am very willing to trade time for this and for example, get assigned some reviews and/or bug fixes in exchange.
Hopefully someone will show up that is willing to trade horses with you here. Good luck!
Cheers
Andrea==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.
And I am not someone who “contribution dumps”. It is true, I am not someone who is very active in the community work either. I am something in between those things. I am always accessible and constructive, and always willing to do volunteer for reviews / provide advise, explanation / fixes / for my code etc if asked. If there is no place for someone like me in this community, I would like to hear that from others as well.
On 29/04/2019 21:40, Niels Charlier wrote:
Again, my CSW PR’s are 80% bug fixes. Trying to sabotage these for personal reasons, well that is on you.
Also, admittedly, they are bug fixes for… my own code. I originally wrote the ISO metadata profile for CSW. But it was apparently never put to use, because the schemes don’t even pas validation, or can be harvested from geonetwork, and bugs were never reported.
So in your own logic, what you’re basically saying is that you’d rather support a faulty, buggy feature that I wrote than an actual functional feature that I wrote.
About your accusations, this is what actually happened:
I did not at all introduce a failing test, I created an offline test environment, so that a whole series of test methods that were part of an online test class, could from then on actually be run with every build. This obviously makes the software more robust and helps to detect regressions. Doing this, exposed that one of these tests mysteriously fails at random. (So to be clear: the only reason that didn’t happen previously was because it was never run. The issue was exposed, not caused, by me.) I didn’t understand the test so I was hoping for some input from the author of the test, who was however not active at that time. But the discussion continued in which I was still involved as well as Kevin, and we both agreed it was better to put that single method in @ignore until the mystery was resolved, since that would still be better than the previous situation before I made the offline test environment and none of the tests were ever run.
You bend the truth to paint an entirely false picture of me, and to excuse your own strange behavior. People that know me, know that I am actually quite a bit perfectionist and code-proud; and rather keep searching for hours and hours to really understand a problem than to introduce an ugly work-around and forget about it. I have never, ever “put up excuses and not do anything” in response to an issue that I caused.
The truth is that you have acted rather hostile to me, for a very, very long time. I have never understood why. Perhaps I annoy you somehow. But nobody else in this community has ever acted this hostile to me. On the other hand, I do think other people have sometimes experienced hostility from your end. You say that things are personal in a community like this. I agree. This community has written down rules, and that is great. But people skills are also import for a community: such as being friendly and nice, not assuming the worst from people, and using constructive communication. Everybody has their own situation, their own background and circumstances, their own strengths and shortcomings, and in any sort of professional environment you have to deal with that.
Regards
Niels
On 29/04/2019 16:52, Andrea Aime wrote:
On Mon, Apr 29, 2019 at 3:36 PM Niels Charlier <niels@anonymised.com> wrote:
What is far more bizarre is that some gets really mad about, but decides to not communicate or report it for months and then launch passive aggressive personal attacks when someone else reports (and fixes) it.
I have lost any faith that you will fix whatever regressions you cause, that’s why I did not communicate, I was sure you’d have put up excuses and not do anything, which would have just made things worse.
Just recently we had a discussion on gwc-devel about a randomly failing test that you introduced, I believe your comment was that you just adapted a test that Gabriel made. No fix was provided on your part, the random failure is still there.Unlike what you seem to suggest, I am pretty sure that all or most of what I have done in the past few years that is not in community modules, are actually just bug fixes. (I guess it is disputable whether making config pages cluster safe is a feature or a bug.) But any major core development has been a long time.
Yes, the large works are a few years old (app-schema, Resource, CSW ISO), but I saw no indication that anything changed in your attitude towards the project and maintenance of your work. Hence my assumption above.
I have done and still do in fact contribute with volunteering work, such as reviews (including for colleagues of yours) and maintenance, including for code I wrote for a company who paid me to develop certain modules or core changes but never intended to pay me to maintain these. I can always be contacted about this work, and I have taken that personal responsibility upon myself entirely, and I am always willing to continue to do so. As far as that certain company is considered, they couldn’t care less if I had developed that stuff and disappeared afterwards, not even being contactable about it.
This is problematic, because if you don’t maintain your contributions to supported modules, someone else will have to, and that someone is a handful of already heavily overloaded core developers, often using significant amounts of their spare time every single week. I think GeoServer as a project has no capacity left to receive “contribution dumps” where the developers provides code and then leaves to do something else.
The other developers contributing a similar amount of commits are actually active in the community with a number
of unpaid activities that help the project stay alive, on the user lists answering questions, on the dev list participating
to discussion, participating to the monthly bug fix code sprint, and cutting releases.
Jody once quite correctly called it “the cost of doing business in a open source project”.Given that your activity is not a charity, but it’s instead paid for, business, it’s only fair to expect the same of you.
Not all major contributors are active any more.
Yes, that is indeed part of the above problem, the ones that are left are more and more overworked, not only from the point of view of the amount of actual coding to do, but more importantly in terms of cognitive load, one thing having to manage 2-5 modules, another entirely is to have to manage and be proficient with several tens of them, I cannot comment about others, but personally I am working beyond my limits.
I cannot really commit to participate in meetings and releases, because I cannot commit to having enough availability at the right time unfortunately to do this as a volunteer because of other obligations. Responding to emails and making reviews as a volunteer is something else entirely, because I can do that at any time, in the middle of the night if need be. After all, my own personal time is the only resource I have for this, and I do the best I can to help the geoserver community. I am not a profit-making business who sells products and services based on geoserver, I am just a very simple freelance programmer on his own. I don’t know if there are many others who have participated an entire week of code sprint purely as a personal volunteer, paying everything except for the flight out of their own pocket, and missing a week’s worth of work in opportunity cost - not in the context of a larger company.
That participation was appreciated, but what keeps the project alive is day to day participation. Think of it as having to keep up with a cow farm, you can show up and do something wonderful for a week long, but if you then leave them for a year and nobody cares about them, by you come back all the cows are dead.
I do these things to do good, not to do evil. I believe in this model of development, and I also wished companies would invest more in the community work. Again, I am not a business, just a simple programmer, and I could easily earn the same living without caring at all about geoserver and its community. This picture you want to paint of me as a leech who takes and profiteers but never cares or gives is just not close to reality. I simply do my best with the resources I have, but I am not perfect.
You are a business, you just happen to be a single person business, but what you do is similar to what GeoSolutions does, we don’t sell a downstream product, simply provide improvement and bug fixing services (so in the end we get paid by the hours worked). And then we dedicate part of the company time, and in the case of some of us, part of personal time, to keep that code going, once the contract is long gone.
Your portion of business relating to GeoServer exists because someone keeps the project alive by answering user questions, doing releases, participating to the monthly bug fix sprints and the like, without being paid to do that.
I understand that you don’t have the same amount of activity and don’t pretend you show up for releases once every two months like we do, or be the most active dev on the user list, or participate at most bug fix code sprints, but at some of those (releases, bug fix sprints, fielding user questions), yes, you should.There is not much need for additional support, because this concerns functionality that geoserver according to the documentation already provides: the iso metadata profile, but which currently really doesn’t work properly. The alternative is to have a new CSW community module with iso metadata profile properly supported. I do think if even the small bug fixes get ignored, this is a personal rather than pragmatic matter. I don’t see how any of the CSW stuff I did can cause problems for anyone.
It is personal, and I’ve said so multiple times in multiple places. Business is indeed not personal, it’s exchanges of money with a service, I don’t have to like the customer. Using spare time is a personal choice instead, and I do all my un-paid reviews on Saturday and Sunday mornings (for the joy of those contributing in paid hours, and wondering why it takes so much time to get one and so much un-synchronized with their needs to get code in).
This makes open source projects something about people, unlike other things like healthcare where one can expect a service in a impersonal way (but only because taxes were paid for it). So yes, personal is the norm in free software communities, you’ll have to learn to deal with it.About ISO support, I did not work on that part of the CSW machinery (only covered dublin core and the generic service engine), in order to review the changes I’d first have to learn about what an ISO record is, and how the code producing it works. That would require even more spare time to be used.
I don’t expect people to do all this for me out of the goodness of their heart, I was in all honesty hoping that people would be able to use this.
I can see the usefulness of the work done (extension of ISO metadata support to cover more elements), I’m simply not willing to take on its review and long term maintenance myself. Again, sorry to sound like a broken record, but if a module is supported, it needs someone to willing to maintain it, adding new code just makes live harder for those that have volunteered to do such maintenance. I volunteer to maintain the code I write, not other people’s code.
Admittedly, I also like to avoid that maintaining forks becomes to much of a complicated matter. I understand the bigger reviews are more tricky, but I am very willing to trade time for this and for example, get assigned some reviews and/or bug fixes in exchange.
Hopefully someone will show up that is willing to trade horses with you here. Good luck!
Cheers
Andrea==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.
_______________________________________________
Geoserver-devel mailing list
[Geoserver-devel@lists.sourceforge.net](mailto:Geoserver-devel@lists.sourceforge.net)
[https://lists.sourceforge.net/lists/listinfo/geoserver-devel](https://lists.sourceforge.net/lists/listinfo/geoserver-devel)
Given that to the extent of my knowledge there’s no prescribed policy on what/how to check for regressions before making a contribution - correct me if I’m wrong so I can follow them myself-, I’d say this is something that could happen to anyone, as demonstrated bellow. I can’t testify for Niels’ speed in reacting to review requests cause I’ve beed away of core geoserver development for too long, but in any case, and since this heated discussion happened on my thread, I’ll give my opinion:
It is very odd to see a discussion like this on what was just intended to be a regular contribution thread, and I think calling someone out like this on a public forum should not be taken lightly. We all know what’s like to be overwhelmed and burnout, and I like to assume we all try to do our best even though sometimes we can piss someone off or not be up to someone else’s expectations.
In any case, for the future I’d recommend any concerns regarding the behavior a any given contributor to be handled by the PSC in a different way. I’m sure there could be, if there isn’t, a psc dedicated mailing list, or something.
So, there are three issues I’ve identified that bothered me:
GEOS-9191’s cause is 5 years old, with the following comment on FileSystemWatcher’s class javadocs: “This implementation currently polls the file system and should be updated with Java 7 WatchService when available.”
#3450 is a regression, or slowdown, call it as you wish, caused by ConfigurationPasswordEncryptionHelper, 7 years old.
#3451, the one that started this heated discussion, is some months old.
The two pull requests above were done in my spare time out of personal interest, and I’m working on GEOS-9191 too, though it probably will need to wait till next weekend for that same reason.
Again, I know having a lot of data on a filesystem data directory is not the norm. Heck it’s not even what it was designed for. But people do that, and good alternatives are lacking.
Actually I started looking at this cause someone at FOSS4G-AR mentioned it took just too long for geoserver to start up with lots of layers.
So, all things considered, maybe we could define a set of pre-release performance regression tests to make sure these things can be identified and addresses earlier?
Cheers,
Gabriel.
Gabriel Roldán
I have been meaning to say what I said for years. I managed to stop myself every time since now, but this time I couldn’t.
It was indeed unprofessional of me, could have handled it better, I apologize.
Cheers
Andrea
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.