Hi,
since I cannot post the logs onto conflunce (times out)
I'll do it here.
-------------------------------------------------------------------------
0) What's up?
1) Releases releases
2) Logs darned logs
3) New DataStore.dispose method
aaime: 0) What's up?
***groldan studies an ugly spec: wgs 1.1
***aaime is coding like a wild cat trying to fix issues before beta4 falls on him
sfarber: what's wgs stand for?
groldan: (wfs 1.1.)
sfarber: ahh, sorry!
groldan: (I'm sorry)
sfarber: somehow hoped there was a web geocoding service spec newer than the last crappy one.
sfarber: saul: porting client apps away from ArcIMS to Geoserver
groldan: yeah!
aaime: go Saul go!!! 
sfarber: Yeah, it's satisfying (if rather rote) work.
groldan: (btw, or my machine is really fast or geoserver become really fast)
***cholmes is on a call about another WFS that takes 20 minutes to return 20mb of GML
aaime: (a mix of the two probaly)
aaime: oh jeez
aaime: with GML2 flat features we take 2 seconds
aaime: with GML3 maybe 6-8seconds
aaime: tell them to go hide themselves :-p
groldan: (would that wfs be doing xslt?)
aaime: Is xslt so heavy?
groldan: could be as heavy as you want
aaime: ha ha
aaime: Ok ok
aaime: let's move to next topic
aaime: 1) Releases releases
aaime: So, 1.6.0beta4 was supposed to go out today but I still see 10 unsolved issues?
aaime: What do we do? (cholmes?)
cholmes: It should go out tomorrow
cholmes: We had to get a demo ready for Mark today
cholmes: But I'll move off the issues and should get it out tomorrow
aaime: Tomorrow means I could try to fix GEOS-1417 too (good :-p)
aaime: Ok, what about 1.5.4?
aaime: Overdue for people hitting 900913, only 4 issues
aaime: (especially now that OL 2.5 is out in the wild)
aaime: cholmes?
cholmes: I think I was going to have groldan take a crack at it
cholmes: probably end of this week
aaime: go groldan go
aaime: That should be all
groldan: would be glad of doing it
aaime: anybody has anything to add?
aaime: (and we would be very grateful of you doing that... me in particular :-p)
aaime: 2) Logs darned logs
groldan: ("that" meaning the release or GEOS-1417?)
aaime: groldan: release
groldan: aaime: phew!
aaime: Ok, we have some disparate user report that altering the logging level does not always work
aaime: I managed to get myself into the situation of not being able to change the log level (but I don't rememebr how)
aaime: today I tried to set the logging level to geotools developer and got no logs in the geoserver.log under the info level...
aaime: so I think there is definitely something going on
aaime: I want to try and investigate this (also to properly answer Martin mails)
sfarber: Good, that means that we can reproduce the problem and therefore fix it!
aaime: well, I cannot reproduce the problem of that user that cannot have log redirected to file for example
aaime: that's nasty
sfarber: Aaime: if you're going to look into this, the places to look at are:
sfarber: org.geoserver.logging.LoggingStartupContextListener.java
sfarber: in web/
groldan: (btw, confluence is still down, I hped to get at the release procedure)
aaime: well, I was planning to sue Sun for including java logging in the jdk but... yeah, I can try your suggestion as well
aaime: really, drop any report of misbehaviours on the devel list
sfarber: I'll take a peek too, and see what I can figure out.
aaime: that is most appreciated
sfarber: The "space in the data dir" problem might be tough to track down on linux, but I'll try that too.
sfarber: The stack trace that guy gave pretty much isolates the problem 100% (if I remember correctly)
sfarber: It was just a URL() blowing up with spaces when log4j tried to initialize to that path
aaime: I work on the "dark side" (xp) so I guess I'll be able to reproduce that one
sfarber: So, seperately from figuring out what's going wrong with the current situation, let's talk about martin's conversation.
sfarber: I see two things:
sfarber: 1) Martin doesn't understand geoserver's logging requirements
aaime: he probably never had to work on a server with multiple webapps running in paralle
sfarber: 2) We have a confusing logging setup.
aaime: you bet we have
sfarber: yeah, that's pretty much the stopper issue.
sfarber: I mean "the stopper fro martin is that he doesn't have multi-webapp use-cases in his head"
aaime: indeed that's a problem
sfarber: I'm a bit fuzzy on our requirements too, however.
sfarber: I know these ones:
sfarber: * Provide a way to not log to stdout
sfarber: * Support different log levels across different geoserver instances
sfarber: * Log to configurable file
sfarber: * Log consistently inside the codebase
aaime: (right, production and testing deployed on the same container... never thought of that one, good one)
aaime: * Change the logging level on the fly
sfarber: What about how we integrate into container logging mechanisms?
sfarber: Yeah, that's a good one.
aaime: We are container agnostic
aaime: Not sure there is any part of the servlet spec speaking about integration with server logging?
sfarber: Ok, so worst case scenario, they set up commons-logging to redirect to their own custom logsystem inside the container or whatever
sfarber: Me neither.
aaime: I guess so
sfarber: Ok, I'm happy with that...
aaime: * make the logging completely configurable by the user
aaime: (see log4j files)
aaime: * have no interfefences from a darn centralized java logging file
groldan: (hmmm if they have the skills to do that, they have it to tweak the geoserver logging config, I'd bet)
aaime: not necessarily, an admin may be comfortable touching a log4j file but maybe not the code
sfarber: do we get interference from the central java logging now?
aaime: Not sure
groldan: there might be tough situations, like wanting to use yet another logging system..
aaime: but since java logging is configured by that dreaded system wide file...
aaime: groldan, commons-logging should take care of that one
sfarber: you can force a different base config with a -D command-line param, I think.
aaime: yep, for the whole VM again
aaime: Quick question for you Saub
aaime: Saul
sfarber: What's up?
aaime: If I set org.geoserver=INFO in the logging configuration
aaime: shouldn't it cascade onto all packages too?
aaime: (I have the impression it does not at the moment, but I'm not sure)
aaime: sorry, onto whatever package matches
aaime: org.geoserver.*
sfarber: Hmm. let me think a sec.
sfarber: Yes, that is the intention
sfarber: It works like this:
sfarber: 1) You create a file with log4j.category.org.geotools=INFO
sfarber: 2) You select that file from the pull-down in geoserver-ui
sfarber: 3) Geoserver calls initLogging() with that selected file, and runs the initLogging code
sfarber: 4) The initlogging code reads the log4j config file, and applies the log-level set by each log4j.category.xxx.xxx line to the java-logging package with the same name
sfarber: er "java logging logger" with the same name
sfarber: Let me track down the lines of code on that one...
aaime: GeoServer.configureGeoServerLogging
sfarber: yeah, that's it.
sfarber: http://svn.codehaus.org/geoserver/trunk/geoserver/main/src/main/java/org/vfny/geoserver/global/GeoServer.java
aaime: so, by the way this thing is configured, we'll never see, say, spring logging right?
sfarber: I'm don't think that's true.
sfarber: You just need to include spring logging in the log4j file that you use to configure geoserver
aaime: but spring stuff is not redirected (or is it)?
aaime: I'm looking at LoggingStartupContextListener
sfarber: Well, spring uses commons-logging already, so there's no need to redirect it.
sfarber: (i think!)
aaime: oh, right
sfarber: I'm pretty sure spring uses commons-logging if it's available, otherwise it tries to use log4j
aaime: Good good
aaime: Ok, anything else or shall we move to next topic?
sfarber: Just a short recap:
sfarber: * Glad to have put down the geoserver logging requirements in written form a bit
sfarber: * I'll take a look at busted geoserver logging, and you should see if there's something weird in XP aaime.
sfarber: * that's all
aaime: Good
sfarber: next topic!
aaime:
aaime: 3) New DataStore.dispose method
aaime: ok, for anyone who wans't on planet geotools in the last week
aaime: we just added a DataStore.dispose method
aaime: that should be call to get rid of whatever resource the datastore is holding onto
aaime: before throwing it away
aaime: basically every time Data (the catalog) is reloaded (as an effect of apply or load)
aaime: the datastores are throw away (this alwasy happened)
aaime: now we do at least dispose them before doing so
jgarnett: sweet
sfarber: great, so that reflection line about ArcSDE is gone now? good!
aaime: no it's not
groldan: its going to be
aaime: those three places where reflection is used, in fact
groldan: once we implement dispose
aaime: but there's a catch
aaime: ArcSDE datastores as of now hold onto shared state
aaime: the connection pool is kept as a static reference into the arcsde connection pool factory
aaime: which on one side is good, because the pool is never re-created
aaime: on the other side is bad, because if you change something in the connection parameters, is not freed either
aaime: (another connection pool is created and stored in the static reference again)
groldan: well, that's not completely true
sfarber: yeah, it's used by the ArcSDE rasters (as a hack) too. ArcSDE rasters 'leech' on an already created connection pool with better params, rather than creating their own.
groldan: I see
aaime: Ah
aaime: So?
groldan: the idea being that a connpool is a "per set of connection parameters" singleton
aaime: In fact I said "if you change the connection parameters" above
aaime: Are the size and other params of an arcsde connection pool modifiable (I'm speaking of tunables, not user/passwd)
groldan: by looking at it, I concede it might leak pools, yes
groldan: at runtime?
aaime: yes
aaime: I mean, like jdbc datastores
groldan: guess not, it was on the wish list though
aaime: you can now change your mind and say, no, I want 20 max connections, not 10
groldan: yeah, wish list
aaime: So it seems to me the decision is:
aaime: * keep arcsde like that (no need to dispose anything since the pool is mostly immutable)
aaime: * avoid the shared state, have a pool created each time an arcsde ds is created, and disposed when it's disposed (like with jdbc datastores)
groldan: * still would be to share between datstore and coverage store
aaime: yeah, the second option makes the sharing problematic...
groldan: sfarber: ideas?
aaime: Is it possible to configure the pool from a container web admin and put it in jndi?
aaime: (like long lived shared object, you know)
groldan: wish so!
sfarber: yeah, I see the issues. I'm not sure I have a great answer though.
aaime: Oh well, let's keep it like this for the moment then?
sfarber: Yeah...maybe we can do some basic tracking to have pools get destroyed when all the datastores that rely on them are disposed()?
aaime: for jdbc datastore it became critical because using DBCP we stopped sharing the pools like arcsde does
aaime: all datastores and all coverage readers
aaime: do coverage readers have a dispose too?
aaime: And besides there is one thing
aaime: when you do "apply" everything gets wiped out
aaime: so by extension, everything using the pool disappears
aaime: (to reappear a fraction of a second later, unless you changed a part of the config that affects it)
sfarber: Hmm, I don't know whether coverage readers have a dispose() method. Gotta check on that.
aaime: Having a config system with direct write (submit and you're done) would allow us to avoid all this mess
aaime: because in that case you would know what changed
aaime: (so you would not try to wipe out everything)
groldan: yeah, problem is that usually you have much less connections available for arcsde than you would have for jdbc
aaime: Oh well, by year 2020 we should have a new config subsystem anyways
groldan: as each one cost money
groldan: so it wouldn't be too ilogical that its not possible to have to connection pools alive at the same time, even if you intend to discard one of them
aaime: I see... yet dropping the pool and recreating it would not change the number of available connection right? The problem is more sharing the pool with raster readers (and performance)
sfarber: yeah, exactly aaime.
aaime: groldan, if you try the following
aaime: connect to arcsde with user1/pw1
aaime: then change the params to user2/pw2
aaime: apply
aaime: and you have lost one pool
aaime: (with the _current_ system)
aaime: I'm wondering how odd that situation is... maybe you have different access limitations according to the user you use to connect to sde no?
groldan: afaik, if you bought 20 connection licenses, you can create 20 simultaneous connections, no matter which user you connect with
groldan: as they're db users after all
aaime: That's not my point
groldan: ah
aaime: what I mean is that if you do the above you'll create two pools
groldan: yes
aaime: and the first one will stay there until geoserver is shut down
aaime: so you've lost connections
groldan: hmmm.. I guess yes, and that's a huge bug
aaime: Again, as I said, this may be a very odd case
groldan: (I mean, the pool closes itself on finalize(), but guess its being held on the factory's static map)
aaime: indeed it is
groldan: an horrible one
sfarber: well, sort-of.
aaime: that's why I was proposing to have one pool per ds
sfarber: The real life use case is this:
aaime: but raster stuff does not allow us to
sfarber: ArcIMS, ArcGis Server and ArcMap itself all 'hog' connections, and often leave 'stale' connections to ArcSDE open all the time.
sfarber: So ArcIMS will crash and leave 20 connections to ArcSDE open (or will continuously open connections over a week period)
aaime: (man...)
sfarber: So most ArcSDE admins are fairly adept at using 'sdemon -o kill -t <pid>'
sfarber: you know?
aaime: I see
sfarber: I actually had to get our SDE admin off my back.
aaime: wy?
sfarber: cause he kept killing the geoserver sde connections when he'd run out of the 128 or 256 limit.
aaime: nice man
sfarber: and I'm like "I KNOW the connections are old...but that's because they are in a pool that actually WORKS!"
groldan: lol
sfarber: anyway, that bug is a bad one, but I think that restarting geoserver to fix it isn't the worst solution in the world.
aaime: Yep, at least until we have the concept of an integrated datastore/coveragestore
aaime: (wouldn't it be nice? ds.getFeatureSource(xxx) and ds.getCoverateReader(yyy))
sfarber: yes, very very nice!
groldan: you can do it under the cover anyway, if needed
aaime: jgarnett, another point for you datastore interface improvement 
sfarber: Hmm. So what's the take-away?
aaime: We do nothing and wait
groldan: ArcSDEResourceStore implements DataStore, CoverageStore
aaime: Well yes, groldan, that was the idea
groldan: (and then delegates, of course)
groldan: yeah, but will that solve the problem we're discussing?
aaime: it would
aaime: because we could keep the datastores and coverage stores in a single list
aaime: and filter only those implementing coveragestore when showing coverage stores
aaime: so you instantiate the vector side and the raster side appears as magic
aaime: too
groldan: as long as we keep the semi-singleton pools and add some sort of reference tracking...
aaime: so it's one entity, one connection pool
aaime: no need for that either
aaime: it's just one object
aaime: the problem with raster readers is that they expect to find the pool in the static holder
sfarber: ahh...wait. There's an idea!
aaime: but if the readers were instantiated from the same object (both vector and raster store)
aaime: then you would not need it
sfarber: hmm...gonna have to think about it asec.