[Geoserver-devel] More issues with date/time data types

Hi,
I'm trying to get our date/time support story more consistent,
and I'm stumbling in a couple of road blocks I'd like to
discuss.

First a little introduction. Up to some time ago, geotools/geoserver
date/time support was quite in disarray. Everything was managed as
a java.util.Date, losing completely the distinction between Date, Time
and Timestamp, which is necessary for good xml encoding (xs:date, xs:time, xs:datetime are different kind of beast).

To keep the distinction around, I hacked the jdbc datastore to keep
on using the sql type, which has the distinction, trying to force
this kind of mapping:
xs:datetime -> java.util.Date or java.sql.Timestamp
xs:date -> java.sql.Date
xs:time -> java.sql.Time

So far so good, but there is still an issue: the timezone.
Calendar seems to be the only class that properly carries around
the timezone, whilst the other classes switch to the current
timezone (the "global time" remains the same, but it's adjusted
to the current timezone).

For example, running the following small java app (requires jaxb in
the path for ISO date/time formatting):

import java.util.Calendar;
import javax.xml.bind.DatatypeConverter;
import com.sun.xml.bind.DatatypeConverterImpl;
public class TimeTest {
     public static void main(String args) {
DatatypeConverter.setDatatypeConverter(DatatypeConverterImpl.theInstance);
         Calendar calendar = DatatypeConverter.parseDateTime("2006-02-27T22:08:12+00:00");
         System.out.println(DatatypeConverter.printDateTime(calendar));
         Calendar c = Calendar.getInstance();
         java.util.Date date = calendar.getTime();
         System.out.println(date);
         c.setTime(date);
         System.out.println(DatatypeConverter.printDateTime(c));
         java.sql.Date sqlDate = new java.sql.Date(date.getTime());
         c.setTime(sqlDate);
         System.out.println(DatatypeConverter.printDateTime(c));
     }
}

on an italian system I get the following:

2006-02-27T22:08:12+00:00
Mon Feb 27 23:08:12 CET 2006
2006-02-27T23:08:12+01:00

(CET being Central European Time). Notice how 22 turned to 23 and
the time zone changed from +00 to +01.

Now, how do we handle this? The timezone does not seem to go thru from
storage to XML encoding (and oh, btw, dbms are timezone aware).
I see three ways:
* find some classes that do keep the timezone intact. Maybe Joda time
   (http://joda-time.sourceforge.net/) can do this.
* accept the loss, and allow the xml encoder to be configured with a
   target timezone (something we can do by injecting a timezone into
   the xml date/time bindings)
* just use the default timezone, if someone needs a different one,
   he can configure the JVM accordingly using user.timezone
   (see http://www.javaworld.com/javaworld/jw-10-2003/jw-1003-time.html?page=4)

The first one is to be checked, but I don't like it because it would
introduce another heavy dependency (joda time jar is around 500kb),
and we would have to fix all datastores to use it instead.

The third one is easy, but it may not work all too well in web
applications, since the system variable setup would be container wide
(anyone knows if in common containers is possible to configure
system variables per-application instead of per-VM?).

The second one seems to strike a nice balance.

Comments, suggestions? Speak up please :slight_smile:
Cheers
Andrea

Its always one thing or another with dates and times isn't it...

As a 4th alternative:

How about always carrying around the date as a Calendar. As soon as the
date comes out of the jdbc driver convert to calendar ( am I correct in
assuming there is no loss of timezone here? ). Then have the mapping to
the correct xs:date, xs:datetime, etc... be done by the xml encoder /
geoserver type mapping. I guess this only works for the new xml encoder
though, not for old gml2 encoder.

As for the other alternatives, I think I like using a time library the
best.

Question: If we use joda can we get rid of jaxb? If so it might just be
worth it.

-Justin

Andrea Aime wrote:

Hi,
I'm trying to get our date/time support story more consistent,
and I'm stumbling in a couple of road blocks I'd like to
discuss.

First a little introduction. Up to some time ago, geotools/geoserver
date/time support was quite in disarray. Everything was managed as
a java.util.Date, losing completely the distinction between Date, Time
and Timestamp, which is necessary for good xml encoding (xs:date,
xs:time, xs:datetime are different kind of beast).

To keep the distinction around, I hacked the jdbc datastore to keep
on using the sql type, which has the distinction, trying to force
this kind of mapping:
xs:datetime -> java.util.Date or java.sql.Timestamp
xs:date -> java.sql.Date
xs:time -> java.sql.Time

So far so good, but there is still an issue: the timezone.
Calendar seems to be the only class that properly carries around
the timezone, whilst the other classes switch to the current
timezone (the "global time" remains the same, but it's adjusted
to the current timezone).

For example, running the following small java app (requires jaxb in
the path for ISO date/time formatting):

import java.util.Calendar;
import javax.xml.bind.DatatypeConverter;
import com.sun.xml.bind.DatatypeConverterImpl;
public class TimeTest {
     public static void main(String args) {
DatatypeConverter.setDatatypeConverter(DatatypeConverterImpl.theInstance);
         Calendar calendar =
DatatypeConverter.parseDateTime("2006-02-27T22:08:12+00:00");
         System.out.println(DatatypeConverter.printDateTime(calendar));
         Calendar c = Calendar.getInstance();
         java.util.Date date = calendar.getTime();
         System.out.println(date);
         c.setTime(date);
         System.out.println(DatatypeConverter.printDateTime(c));
         java.sql.Date sqlDate = new java.sql.Date(date.getTime());
         c.setTime(sqlDate);
         System.out.println(DatatypeConverter.printDateTime(c));
     }
}

on an italian system I get the following:

2006-02-27T22:08:12+00:00
Mon Feb 27 23:08:12 CET 2006
2006-02-27T23:08:12+01:00
2006-02-27T23:08:12+01:00

(CET being Central European Time). Notice how 22 turned to 23 and
the time zone changed from +00 to +01.

Now, how do we handle this? The timezone does not seem to go thru from
storage to XML encoding (and oh, btw, dbms are timezone aware).
I see three ways:
* find some classes that do keep the timezone intact. Maybe Joda time
   (http://joda-time.sourceforge.net/) can do this.
* accept the loss, and allow the xml encoder to be configured with a
   target timezone (something we can do by injecting a timezone into
   the xml date/time bindings)
* just use the default timezone, if someone needs a different one,
   he can configure the JVM accordingly using user.timezone
   (see
http://www.javaworld.com/javaworld/jw-10-2003/jw-1003-time.html?page=4)

The first one is to be checked, but I don't like it because it would
introduce another heavy dependency (joda time jar is around 500kb),
and we would have to fix all datastores to use it instead.

The third one is easy, but it may not work all too well in web
applications, since the system variable setup would be container wide
(anyone knows if in common containers is possible to configure
system variables per-application instead of per-VM?).

The second one seems to strike a nice balance.

Comments, suggestions? Speak up please :slight_smile:
Cheers
Andrea

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

!DSPAM:4007,470b477e285954901796417!

--
Justin Deoliveira
The Open Planning Project
http://topp.openplans.org

Justin Deoliveira ha scritto:

Its always one thing or another with dates and times isn't it...

As a 4th alternative:

How about always carrying around the date as a Calendar. As soon as the
date comes out of the jdbc driver convert to calendar ( am I correct in
assuming there is no loss of timezone here? ). Then have the mapping to
the correct xs:date, xs:datetime, etc... be done by the xml encoder /
geoserver type mapping. I guess this only works for the new xml encoder
though, not for old gml2 encoder.

I think we can do that, but we would need to create new attribute types,
TemporalAttributeType is just too generic (or as an alternative, have a
subtype information as a field). Then we would need to change the
AttributeTypeFactory.newAttributeType to allow a way to specify
the date/time flavour.
Finally all datastores should be updated as a consequence.
Also consider I'm talking of changes to be performed against gt2 2.4.x.

As for the other alternatives, I think I like using a time library the
best.

Question: If we use joda can we get rid of jaxb? If so it might just be
worth it.

I'm under the impression you're using at least another jaxb class... was
it something to handle namespaces? I just can't rember.
Yes, it seem Joda can handle those formats, at least there
is an XStream extension converter that uses it to handle ISO 8601
formats, see here: http://jira.codehaus.org/browse/XSTR-116

The other thing is that using Joda time classes we would make gt2-main
depend on it (the feature model is there), and that would be a new
and sizeable dependency. What do gt2 developers think?
In any case, it seems to me that such a sizeable switch could be
performed on trunk, but not in an RC of 2.4.x...
That's why I was sponsoring option 2, it requires very few changes :wink:

Cheers
Andrea

Justin’s solution seems the most elegant - i.e. this is simply an encoding issue at both the source and XML output end.

The other option we have in the community-schemas stuff is to bind a node to a specfic data type, which is something we will always need to do where schemas use generic types but we want to implement a specifi type (eg a numerical date positon, not a period, or even the name of a era “Jurassic” for a GM_TimeObject).

Given the need for that capability, I’d vote against doing too much on a type-specific hack just now, but moving to the most capable generic type (eg Calendar - though I’m not too expert in its use I admit) makes sense.

The other thing to think about of course is profiling performance - we know that time will be a heavily used attribute, with very large numbers of results when handling time-series, and nearly always used as a query axis in those case.

For myself, I would suggest we live with the ugly limitations of date in 2.4, and would push for a proper solution on trunk rather than a lot of effort on 2.4 which will just delay and derail the real progress required.

i.e. perhaps Andreas solution #3 will work for 2.4 - its a system wide config, for expert users, with a proper solution in the pipeline.

I think we would be kidding ourselves that a lot of work on a specific case is not going to result in making the next step less stable.

Rob

On 10/10/07, Justin Deoliveira <jdeolive@anonymised.com> wrote:

Its always one thing or another with dates and times isn’t it…

As a 4th alternative:

How about always carrying around the date as a Calendar. As soon as the
date comes out of the jdbc driver convert to calendar ( am I correct in
assuming there is no loss of timezone here? ). Then have the mapping to
the correct xs:date, xs:datetime, etc… be done by the xml encoder /
geoserver type mapping. I guess this only works for the new xml encoder
though, not for old gml2 encoder.

As for the other alternatives, I think I like using a time library the
best.

Question: If we use joda can we get rid of jaxb? If so it might just be
worth it.

-Justin

Andrea Aime wrote:

Hi,
I’m trying to get our date/time support story more consistent,
and I’m stumbling in a couple of road blocks I’d like to
discuss.

First a little introduction. Up to some time ago, geotools/geoserver
date/time support was quite in disarray. Everything was managed as
a java.util.Date, losing completely the distinction between Date, Time
and Timestamp, which is necessary for good xml encoding (xs:date,
xs:time, xs:datetime are different kind of beast).

To keep the distinction around, I hacked the jdbc datastore to keep
on using the sql type, which has the distinction, trying to force
this kind of mapping:
xs:datetime → java.util.Date or java.sql.Timestamp
xs:date → java.sql.Date
xs:time → java.sql.Time

So far so good, but there is still an issue: the timezone.
Calendar seems to be the only class that properly carries around
the timezone, whilst the other classes switch to the current
timezone (the “global time” remains the same, but it’s adjusted
to the current timezone).

For example, running the following small java app (requires jaxb in
the path for ISO date/time formatting):

import java.util.Calendar;
import javax.xml.bind.DatatypeConverter;
import com.sun.xml.bind.DatatypeConverterImpl;
public class TimeTest {
public static void main(String args) {
DatatypeConverter.setDatatypeConverter(DatatypeConverterImpl.theInstance);
Calendar calendar =
DatatypeConverter.parseDateTime(“2006-02-27T22:08:12+00:00”);
System.out.println (DatatypeConverter.printDateTime(calendar));
Calendar c = Calendar.getInstance();
java.util.Date date = calendar.getTime();
System.out.println(date);
c.setTime (date);
System.out.println(DatatypeConverter.printDateTime(c));
java.sql.Date sqlDate = new java.sql.Date(date.getTime());
c.setTime(sqlDate);
System.out.println (DatatypeConverter.printDateTime(c));
}
}

on an italian system I get the following:

2006-02-27T22:08:12+00:00
Mon Feb 27 23:08:12 CET 2006
2006-02-27T23:08:12+01:00
2006-02-27T23:08:12+01:00

(CET being Central European Time). Notice how 22 turned to 23 and
the time zone changed from +00 to +01.

Now, how do we handle this? The timezone does not seem to go thru from
storage to XML encoding (and oh, btw, dbms are timezone aware).
I see three ways:

The first one is to be checked, but I don’t like it because it would
introduce another heavy dependency (joda time jar is around 500kb),
and we would have to fix all datastores to use it instead.

The third one is easy, but it may not work all too well in web
applications, since the system variable setup would be container wide
(anyone knows if in common containers is possible to configure
system variables per-application instead of per-VM?).

The second one seems to strike a nice balance.

Comments, suggestions? Speak up please :slight_smile:
Cheers
Andrea


This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/


Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

!DSPAM:4007,470b477e285954901796417!


Justin Deoliveira
The Open Planning Project
http://topp.openplans.org


This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/


Geotools-devel mailing list
Geotools-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel

Rob Atkinson ha scritto:

Justin's solution seems the most elegant - i.e. this is simply an encoding issue at both the source and XML output end.

I agree it's elegant not too sure I'd call it a pure xml encoding issue thought. People are going to use those Calendar objects
to build applications (since that's what we're going to have inside
the Features). Using the java.util.Date hierarchy has the
advantage of being familiar to most java programmers (more familiar
than Calendar, that is).

I also have the feeling that doing resultSet.getObject(xxx) on a
time column will return a java.sql.Time, if it does, then the original
timezone information is lost anyways on databases.
This hibernate article seems to suggest so: http://www.hibernate.org/100.html
That is, the impression is that you can do things so that you read
date/time/timestamp in whatever timezone you want, but you can't
get to know what the db timezone is. I'll investigate further during
the weekend.

Even if this cuts off databases, keeping around the timezone information
can be worthy in the case of cascaded WFS, where we get to parse the
XML and thus retain the original timezone info if using Calendar objects.

For myself, I would suggest we live with the ugly limitations of date in 2.4, and would push for a proper solution on trunk rather than a lot of effort on 2.4 which will just delay and derail the real progress required.

i.e. perhaps Andreas solution #3 will work for 2.4 - its a system wide config, for expert users, with a proper solution in the pipeline.

I believe it is. Yet, it seems I failed to convey the difference between
a short term solution that can be done now, and a long term solution
that I fear won't be applied at all unless there is specific funding
for it. There is no sponsoring behind this topic, it's just something
I felt was important because it's one of the basics we don't get right.

Then again, the general consensus seems to be to go for the clean way.
I hope we'll have the resources (and will) to address it in the not too distant future, in other words, that it does not become another GeoAPI filter thing (everybody agrees we should clean up the code base from
the old filters, yet nobody is doing anything about it because it's too
much work).

I think we would be kidding ourselves that a lot of work on a specific case is not going to result in making the next step less stable.

Not sure I understand? :slight_smile:
Cheers
Andrea