[Geoserver-devel] XStreamPersister.SRSConverter taking a lot of time

GeoServer GeoServer Developer
,
1

Hi,
investigating why saving a feature type was taking so long
on trunk I found out this code:

@Override
public String toString(Object obj) {
     CoordinateReferenceSystem crs = (CoordinateReferenceSystem) obj;
     try {
         return "EPSG:" + CRS.lookupEpsgCode(crs, true);
     }
     catch (FactoryException e) {
         XStreamPersister.LOGGER.warning( "Could not determine epsg code of crs, encoding as WKT");
         return crs.toWKT();
     }
}

Doing a full lookup can take a whole lot of time, can't we
use lookupEpsgCode(crs, false) instead, or just save the
wkt form? In any case we're saving the native CRS there,
I don't believe any significant information will be lost
if we just use the WKT form (and if the native code came
from a EPSG id, like those coming from postgis, the
lookupEspgCode(crs, false) will quickly catch the official
code anyways).

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

2

Unfortunately I have run into this issue in a few places, and the only time the answer is to do a full lookup. I always to try to set fullScan to false, but it seems to only work some of the time. I understand the cost of the lookup but I see no other alternative in a case where you *need* an SRS code.

Can we perhaps on startup of geoserver force a full lookup and cache it so that we can be 100% sure that lookupEpsgCode with fullScan set to false will return an answer?

I could amend the code to use the WKT, but that would be a bit of a pain. It will also be problematic with rest, because an EPSG code is needed in order to get at data. Making the client parse WKT will be problematic.

Andrea Aime wrote:

Hi,
investigating why saving a feature type was taking so long
on trunk I found out this code:

@Override
public String toString(Object obj) {
     CoordinateReferenceSystem crs = (CoordinateReferenceSystem) obj;
     try {
         return "EPSG:" + CRS.lookupEpsgCode(crs, true);
     }
     catch (FactoryException e) {
         XStreamPersister.LOGGER.warning( "Could not determine epsg code of crs, encoding as WKT");
         return crs.toWKT();
     }
}

Doing a full lookup can take a whole lot of time, can't we
use lookupEpsgCode(crs, false) instead, or just save the
wkt form? In any case we're saving the native CRS there,
I don't believe any significant information will be lost
if we just use the WKT form (and if the native code came
from a EPSG id, like those coming from postgis, the
lookupEspgCode(crs, false) will quickly catch the official
code anyways).

Cheers
Andrea

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

3

Justin Deoliveira ha scritto:

Unfortunately I have run into this issue in a few places, and the only time the answer is to do a full lookup. I always to try to set fullScan to false, but it seems to only work some of the time. I understand the cost of the lookup but I see no other alternative in a case where you *need* an SRS code.

Can we perhaps on startup of geoserver force a full lookup and cache it so that we can be 100% sure that lookupEpsgCode with fullScan set to false will return an answer?

Nope, that would not help. The fast version works only if your
random WKT contains the AUTHORITY specification with the code
the wkt is supposed to be, in that case the fast path reads
the official code and compares the two, returning the official
one if equal

I could amend the code to use the WKT, but that would be a bit of a pain. It will also be problematic with rest, because an EPSG code is needed in order to get at data. Making the client parse WKT will be problematic.

I've lost you there. The SRS is a string and is stored so should
be always available anyways, what's taking long is trying to represent
the native CRS as an epsg code (at least, that is my guess... I'll
check better). No OGC request uses the native CRS no?

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

4

Andrea Aime wrote:

Justin Deoliveira ha scritto:

Unfortunately I have run into this issue in a few places, and the only time the answer is to do a full lookup. I always to try to set fullScan to false, but it seems to only work some of the time. I understand the cost of the lookup but I see no other alternative in a case where you *need* an SRS code.

Can we perhaps on startup of geoserver force a full lookup and cache it so that we can be 100% sure that lookupEpsgCode with fullScan set to false will return an answer?

Nope, that would not help. The fast version works only if your
random WKT contains the AUTHORITY specification with the code
the wkt is supposed to be, in that case the fast path reads
the official code and compares the two, returning the official
one if equal

Then I am not sure I can avoid an expensive lookup. If someone gives me a shapefile with no authority in its .prj what can I do?

I could amend the code to use the WKT, but that would be a bit of a pain. It will also be problematic with rest, because an EPSG code is needed in order to get at data. Making the client parse WKT will be problematic.

I've lost you there. The SRS is a string and is stored so should
be always available anyways, what's taking long is trying to represent
the native CRS as an epsg code (at least, that is my guess... I'll
check better). No OGC request uses the native CRS no?

I was referring to saving out the WKT of the CRS, rather that first looking up the epsg code and then saving that out.

I am also curious as to under what circumstances a look up can be as expensive as you refer to. A quick test of a lookup with full scan of some random WKT I time at ~800ms on my machine. The non full scan takes ~650ms. While I realize this adds up in a loop that has to iterate many times. Is that where you are running up against it?

Regardless, I am not against adding a flag to disable the lookup if that will solve the problem, as long as we have the option to enable it.

Cheers
Andrea

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

5

Justin Deoliveira ha scritto:

Andrea Aime wrote:

Justin Deoliveira ha scritto:

Unfortunately I have run into this issue in a few places, and the only time the answer is to do a full lookup. I always to try to set fullScan to false, but it seems to only work some of the time. I understand the cost of the lookup but I see no other alternative in a case where you *need* an SRS code.

Can we perhaps on startup of geoserver force a full lookup and cache it so that we can be 100% sure that lookupEpsgCode with fullScan set to false will return an answer?

Nope, that would not help. The fast version works only if your
random WKT contains the AUTHORITY specification with the code
the wkt is supposed to be, in that case the fast path reads
the official code and compares the two, returning the official
one if equal

Then I am not sure I can avoid an expensive lookup. If someone gives me a shapefile with no authority in its .prj what can I do?

That SRS xstream converter falls back on the WKT representation
when anything else fails no?
It also means the REST client will have to be ready and parse both
syntaxes no?

I could amend the code to use the WKT, but that would be a bit of a pain. It will also be problematic with rest, because an EPSG code is needed in order to get at data. Making the client parse WKT will be problematic.

I've lost you there. The SRS is a string and is stored so should
be always available anyways, what's taking long is trying to represent
the native CRS as an epsg code (at least, that is my guess... I'll
check better). No OGC request uses the native CRS no?

I was referring to saving out the WKT of the CRS, rather that first looking up the epsg code and then saving that out.

You said:

> I could amend the code to use the WKT, but that would be a bit of a
> pain. It will also be problematic with rest, because an EPSG code is
> needed in order to get at data. Making the client parse WKT will be
> problematic.

The client will have to be ready and parse WKT anyways, even the
extensive lookup can fail, in that case the SRSConverter will use
WKT syntax. That's why I was suggesting to just use the WKT converter
or at least to avoid the expensive scan.

When you say "the EPSG code is needed in order to get at data" I think
you mean when using OGC services? But in that case you have to use
the declared CRS that is stored in the SRS code, not the native
CRS, no?

I am also curious as to under what circumstances a look up can be as expensive as you refer to. A quick test of a lookup with full scan of some random WKT I time at ~800ms on my machine. The non full scan takes ~650ms. While I realize this adds up in a loop that has to iterate many times. Is that where you are running up against it?

Nope, as stated in my first mail, I'm in the order of seconds.
The full lookup goes through 3 stages:
- fast lookup by authority
- quick query against the EPSG database to find a handful of possible
   candidates (non exaustive list)
- full scan against any and all codes contained in the EPSG database

The code that I was trying to lookup is a British Columbia one, see
the attached code sample. If I try to run that from GeoServer it
blows up due to one of our extra custom definitions (which is lacking the authority bit), if you run it inside Geotools, or remove our
extra definitions, on my PC, it reports a time of 6 seconds, and then will return nothing, as an equivalent official code cannot be found.

I also made a little test as to whether a successful lookup, even in full scan mode, is a common occurrence or not.
Since I've started working for GeoServer I received quite a bit of
sample data from users, so I have a collection of around 350 .prj files
sitting on my disk. I made a quick app that opens all of them
and reports for each how much time it took and whether the lookup
was successful or not.
To make it quicker I made sure only wkt that were really different
were taken into consideration (but GeoServer won't have this luxury,
the results of a failed lookup are not cached, so if it takes 4
seconds to fail to lookup a code those 4 seconds will be paid over
and over... something I may try to fix in referencing btw).

The results were interesting. Out of 360 prj files I had in fact only 71
different WKT representations, out of which:
32 looked up to a EPSG code successfully
32 failed to lookup but did not throw an exception in the process
7 that did throw an exception in the process

Those that failed to lookup took usually around 3-4 seconds to
report the null epsg code (that is, lookup failure). This is
better than the 6 above because the server VM is hot after
a few lookups.

So if the data set sitting on my disk is any indication, the
SRSConverter against real world data will be forced to return
the WKT representation in more than 50% of the cases.

I also attached the application and my set of .prj files so
that you can double check on your machine, and eventually
try with whatever other set of prjs you have handy
("find . -name "*.prj" -exec cp {} /tmp/prj \;"
  is your friend if you want to collect all prjs on your fs
  in a single folder)

Also, from what I can see looking in the generated featureType.xml,
the WKT definition is used anyways for the native CRS, what is
forcing the lookup is possibly the native BBOX (did not fully
investigate).

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

(attachments)

WKTDecoder.java (1008 Bytes)
prj.tar.bz2 (8.46 KB)
FullLookups.java (2.09 KB)
featuretype.xml (4.04 KB)

6

Andrea Aime wrote:

Justin Deoliveira ha scritto:

Andrea Aime wrote:

Justin Deoliveira ha scritto:

Unfortunately I have run into this issue in a few places, and the only time the answer is to do a full lookup. I always to try to set fullScan to false, but it seems to only work some of the time. I understand the cost of the lookup but I see no other alternative in a case where you *need* an SRS code.

Can we perhaps on startup of geoserver force a full lookup and cache it so that we can be 100% sure that lookupEpsgCode with fullScan set to false will return an answer?

Nope, that would not help. The fast version works only if your
random WKT contains the AUTHORITY specification with the code
the wkt is supposed to be, in that case the fast path reads
the official code and compares the two, returning the official
one if equal

Then I am not sure I can avoid an expensive lookup. If someone gives me a shapefile with no authority in its .prj what can I do?

That SRS xstream converter falls back on the WKT representation
when anything else fails no?
It also means the REST client will have to be ready and parse both
syntaxes no?

I could amend the code to use the WKT, but that would be a bit of a pain. It will also be problematic with rest, because an EPSG code is needed in order to get at data. Making the client parse WKT will be problematic.

I've lost you there. The SRS is a string and is stored so should
be always available anyways, what's taking long is trying to represent
the native CRS as an epsg code (at least, that is my guess... I'll
check better). No OGC request uses the native CRS no?

I was referring to saving out the WKT of the CRS, rather that first looking up the epsg code and then saving that out.

You said:

> I could amend the code to use the WKT, but that would be a bit of a
> pain. It will also be problematic with rest, because an EPSG code is
> needed in order to get at data. Making the client parse WKT will be
> problematic.

The client will have to be ready and parse WKT anyways, even the
extensive lookup can fail, in that case the SRSConverter will use
WKT syntax. That's why I was suggesting to just use the WKT converter
or at least to avoid the expensive scan.

I am thinking lightweight clients, like javascript, which probably do not have the capability to parse the WKT. Anyways, looking at what actually happens when a feature type is persisted, the nativeCRS is not used to get an EPSG code. The property ResourceInfo.getSRS(), which is just a simple string, is used.

The epsg code lookup seems to happen when encoding the bounds, native and latlon of the resource. And I guess for the native bounds this is wrong behavior since as you illustrated, much of the time this will be null. And for the lat lon bounding box the information is somewhat redundant.

So amending the envelope converter should remove any extensive lookups from the feature type encoding pipeline. I still think their will be a bit of info missing from a client trying to access OGC services pov, for instance in cases where a feature type is being reprojected on the fly, but I will worry about that when said case comes up.

When you say "the EPSG code is needed in order to get at data" I think
you mean when using OGC services? But in that case you have to use
the declared CRS that is stored in the SRS code, not the native
CRS, no?

Yes, I mean that a client will have an easier time accessing the WMS if it has the epsg code right? I mean it could get the resource, get the WMS capabilities, look up the layer SRS, then make a getMap, but it is one extra step. But I am not in the business of writing clients so this argument probably does not hold much weight, would be interesting to ask one of the javascript folks what they think.

I am also curious as to under what circumstances a look up can be as expensive as you refer to. A quick test of a lookup with full scan of some random WKT I time at ~800ms on my machine. The non full scan takes ~650ms. While I realize this adds up in a loop that has to iterate many times. Is that where you are running up against it?

Nope, as stated in my first mail, I'm in the order of seconds.
The full lookup goes through 3 stages:
- fast lookup by authority
- quick query against the EPSG database to find a handful of possible
  candidates (non exaustive list)
- full scan against any and all codes contained in the EPSG database

The code that I was trying to lookup is a British Columbia one, see
the attached code sample. If I try to run that from GeoServer it
blows up due to one of our extra custom definitions (which is lacking the authority bit), if you run it inside Geotools, or remove our
extra definitions, on my PC, it reports a time of 6 seconds, and then will return nothing, as an equivalent official code cannot be found.

I also made a little test as to whether a successful lookup, even in full scan mode, is a common occurrence or not.
Since I've started working for GeoServer I received quite a bit of
sample data from users, so I have a collection of around 350 .prj files
sitting on my disk. I made a quick app that opens all of them
and reports for each how much time it took and whether the lookup
was successful or not.
To make it quicker I made sure only wkt that were really different
were taken into consideration (but GeoServer won't have this luxury,
the results of a failed lookup are not cached, so if it takes 4
seconds to fail to lookup a code those 4 seconds will be paid over
and over... something I may try to fix in referencing btw).

The results were interesting. Out of 360 prj files I had in fact only 71
different WKT representations, out of which:
32 looked up to a EPSG code successfully
32 failed to lookup but did not throw an exception in the process
7 that did throw an exception in the process

Those that failed to lookup took usually around 3-4 seconds to
report the null epsg code (that is, lookup failure). This is
better than the 6 above because the server VM is hot after
a few lookups.

So if the data set sitting on my disk is any indication, the
SRSConverter against real world data will be forced to return
the WKT representation in more than 50% of the cases.

I also attached the application and my set of .prj files so
that you can double check on your machine, and eventually
try with whatever other set of prjs you have handy
("find . -name "*.prj" -exec cp {} /tmp/prj \;"
is your friend if you want to collect all prjs on your fs
in a single folder)

Also, from what I can see looking in the generated featureType.xml,
the WKT definition is used anyways for the native CRS, what is
forcing the lookup is possibly the native BBOX (did not fully
investigate).

Cheers
Andrea

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.