[Geoserver-users] CITE tests

GeoServer GeoServer User
1

Hi,

I have installed GeoServer and would be interested to run the OGC CITE tests for WMS 1.1.1 against my GeoServer instance.
I have installed the test data (btw, I cannot preview in OpenLayers some of the features, e.g. ponds...) and can start the tests, but execution takes forever, perhaps due to the large number of SRS supported by GeoServer.

Is it possible to disable/delete some of the SRS's, to speed up testing?
A tutorial on verifying GeoServer claimed compliance with OGC standards would be useful (apparently there was something similar in the old documentation...)

Best regards, thanks for helping,
LB

---
Dott. Lorenzo Bigagli
Istituto di Metodologie per l'Analisi Ambientale
del Consiglio Nazionale delle Ricerche (IMAA-CNR)

i: Area della Ricerca di Potenza, Contrada Santa Loja
   Zona Industriale, 85050 Tito Scalo (PZ), Italia
t: +39 0971 427221
f: +39 0971 427222
m: lorenzo.bigagli@anonymised.com

2

On 31/05/10 16:57, Lorenzo Bigagli wrote:

A tutorial on verifying GeoServer claimed compliance with OGC standards would be useful (apparently there was something similar in the old documentation...)

The procedure for performing CITE testing is included in the GeoServer developer manual. CITE tests are performed as part of the GeoServer release process:
http://docs.geoserver.org/trunk/en/developer/release-guide/index.html#cite-testing
http://docs.geoserver.org/trunk/en/developer/cite-test-guide/index.html#cite-test-guide

My colleague Victor Tey encountered some problems on Windows setting up the test machinery for the recent GeoServer 2.0.2 release; he will be updating the CITE test documentation. Some of the documentation may refer to older releases, but the process is similar.

Thanks to recent work by Justin Deoliveira, CITE testing is now automated and run at least nightly:
http://hudson.opengeo.org/hudson/view/cite/

For more information on CITE testing, please refer to the GeoServer developer mailing list (and its archive).

Kind regards,

--
Ben Caradoc-Davies <Ben.Caradoc-Davies@anonymised.com>
Software Engineering Team Leader
CSIRO Earth Science and Resource Engineering
Australian Resources Research Centre

3

Lorenzo Bigagli ha scritto:

Hi,

I have installed GeoServer and would be interested to run the OGC
CITE tests for WMS 1.1.1 against my GeoServer instance. I have
installed the test data (btw, I cannot preview in OpenLayers some of
the features, e.g. ponds...) and can start the tests, but execution
takes forever, perhaps due to the large number of SRS supported by
GeoServer.

Hum... odd? We do run the CITE tests nightly using a local CITE engine
and the whole processing time is around one minute (actually, not exactly the official ones, as we're running against svn checkouts of the
various source codes, meaning it's actually a newer version than the
online one):

http://hudson.opengeo.org/hudson/view/cite/

Is it possible to disable/delete some of the SRS's, to speed up
testing? A tutorial on verifying GeoServer claimed compliance with
OGC standards would be useful (apparently there was something similar
in the old documentation...)

As far as I remember the WMS ones are the easiest to run, just
run GeoServer with the WMS directory provided in svn and
point the CITE engine to your GeoServer instance (which must be
running on a publicly accessible server):

http://docs.geoserver.org/latest/en/developer/cite-test-guide/index.html#run-wms-1-1-tests

Btw, just started GS and displayed points with success (screenshot
attached). But I'm not using a released version, running from the
current sources instead.

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

pond.png

4

Hi and thanks to all for the prompt reply.

I just updated to GS 2.0.2 (I've seen there were changes wrt CITE, too) and I'm trying to replicate your results.
I'm using a MacOS X 10.6 machine, installing the WMS 1.1.1 test data downloadable from the CITE website and running the official TEAM Engine test suite.
So I reckon the problem could be worked around running the patched engine/tests from localhost, as by the documentation.

I was wondering on the "strict CITE compliance" option: what's the use of it?

Eventually, I confirm that I can't preview the ponds dataset. I receive the attached ServiceException: does it help to clarify?

Thanks again,
  LB

Ponds_preview_response.txt (398 Bytes)

5

Lorenzo Bigagli ha scritto:

Hi and thanks to all for the prompt reply.

I just updated to GS 2.0.2 (I've seen there were changes wrt CITE,
too) and I'm trying to replicate your results.

The official test data is just a reference afaik, there is no mandate
to actually run against it. We have our own copy:
http://svn.codehaus.org/geoserver/trunk/data/citewms-1.1/

The problem is mostly likely due to you trying to use the
same data provided by OGC

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

6

Lorenzo Bigagli ha scritto:

Hi Andrea,

Please keep answers on the list

to me it makes much sense that the official test data are the ones to
use when officially testing the compliance, doesn't it? Unless they
are bugged, of course. As I understand, the compliance program of OGC
is undergoing some major management restructuring, what may hopefully
clarify the situation.

Nope, certification wise it makes no more sense than using your own
set derived from the official one. Here is why.
When you do the certification you just point the engine to the capabilities document of your server, it runs the requests, checks
the responses, if you pass that you can submit your certification
request (and pay for it).
At no point there is any check that you're running against the actual
sample data, nor there is a requirement to do so.

However, I have tried downloading your own copy of the test data, but
I experience the same problem (see log attached). Should I patch the
Ponds dataset somehow, before trying to preview?

Ah, the zero envelope issue. I think Justin fixed it recently... are you
sure you're running against a recent svn checkout from the url
I pointed you at?

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

7

Hi,

ok, I agree on your explanation about test data.

And actually I am running the stable 2.0.2, so most probably it doesn’t include Justin’s fix.
Clear enough.

For the sake of completeness, I have recovered a screenshot of a test session I ran a few days ago (see picture attached).
From that point on, there is a long list of:
Test wms:wmsops-getmap-each-srs (View Details): Passed/Failed

I interrupted the test after some time.
I reckon you are running the tests from localhost, nevertheless I would doubt that so many tests (and many more others before and after those in the picture) can be executed in one minute.
Are you sure your test cases are equivalent to the official ones?

Regards,
LB

Il giorno 31/mag/2010, alle ore 12.52, Andrea Aime ha scritto:

Lorenzo Bigagli ha scritto:

Hi Andrea,

Please keep answers on the list

to me it makes much sense that the official test data are the ones to

use when officially testing the compliance, doesn’t it? Unless they

are bugged, of course. As I understand, the compliance program of OGC

is undergoing some major management restructuring, what may hopefully

clarify the situation.

Nope, certification wise it makes no more sense than using your own
set derived from the official one. Here is why.
When you do the certification you just point the engine to the capabilities document of your server, it runs the requests, checks
the responses, if you pass that you can submit your certification
request (and pay for it).
At no point there is any check that you’re running against the actual
sample data, nor there is a requirement to do so.

However, I have tried downloading your own copy of the test data, but

I experience the same problem (see log attached). Should I patch the

Ponds dataset somehow, before trying to preview?

Ah, the zero envelope issue. I think Justin fixed it recently… are you
sure you’re running against a recent svn checkout from the url
I pointed you at?

Cheers
Andrea

–
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

Dott. Lorenzo Bigagli
Istituto di Metodologie per l’Analisi Ambientale
del Consiglio Nazionale delle Ricerche (IMAA-CNR)

i: Area della Ricerca di Potenza, Contrada Santa Loja
Zona Industriale, 85050 Tito Scalo (PZ), Italia
t: +39 0971 427221
f: +39 0971 427222
m: lorenzo.bigagli@anonymised.com

8

Lorenzo Bigagli ha scritto:

Hi,

ok, I agree on your explanation about test data.

And actually I am running the stable 2.0.2, so most probably it doesn't include Justin's fix.

The fix is actually in the layer configuration. Just make its bounding
box a little larger so that it's not empty.

Clear enough.

For the sake of completeness, I have recovered a screenshot of a test session I ran a few days ago (see picture attached).
From that point on, there is a _long_ list of:
Test wms:wmsops-getmap-each-srs <http://cite.opengeospatial.org/teamengine/listings/~2Flocal~2Fsrv~2Ftomcat~2Fwebapps~2Fteamengine~2FWEB-INF~2Fscripts~2Fwms-1.1.1-r1~2Fctl~2Fwms.html#wms:wmsops-getmap-each-srs&gt; (View Details <http://cite.opengeospatial.org/teamengine/viewTestLog.jsp?test=s0002/d207e2103_4&gt;\): Passed/Failed

I interrupted the test after some time.
I reckon you are running the tests from localhost, nevertheless I would doubt that so many tests (and many more others before and after those in the picture) can be executed in one minute.

Ah, that one is new to me, last time I run the official tests (was it
one year go? not sure) there was no loop going over all the declared
srs.
Given what GS declares by default that would generate roughtly
4000 * num_layer request, which is definitely not the case of my
last run (the tests were something like 120).

Are you sure your test cases are equivalent to the official ones?

I haven't been involved in CITE testing for quite a bit of time
so no, I'm not sure. Justin should be able to provide more information,
he automated the CITE test suites we're running each night only
some time ago.

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

9

Andrea Aime ha scritto:

I haven't been involved in CITE testing for quite a bit of time
so no, I'm not sure. Justin should be able to provide more information,
he automated the CITE test suites we're running each night only
some time ago.

Btw, just had a quick look. We're now running the CITE tests as they
come from the OGC private svn as of revision 2740 (one needs a OGC
portal username/password to access it).

The version control is now at revision 3331, though only 4 commits
happened on the wms 1.1 directory after 2740. Looking at the diff
I can see there are substantial changes (the diff, as obtained
with https://svn.opengeospatial.org:8443/ogc-projects/cite/scripts/wms/1.1.1/
is 1.2MB!!!).

That said, I know no way to guess which version the official CITE
engine is using, and when the official engine changed its tests
into the current ones (they are not very public about either of them).

Long story short:
- we do what we can to keep GS compliant (running CITE nightly is
   more than many other do)
- we don't have the resources to actually run the official tests
   for each release, the official engine is slow and unreliable
   and a constantly moving target (as I said, they change it
   without warning)
- it seems the official engine made a jump since we last
   decided with svn revision to use... do you have time to ask
   in OGC what version they are now using?

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

10

Andrea Aime ha scritto:

I interrupted the test after some time.
I reckon you are running the tests from localhost, nevertheless I would doubt that so many tests (and many more others before and after those in the picture) can be executed in one minute.

Btw, there is a simple workaround for this one: go into the
WMS configuration and declare a limited list of EPSG codes you
want to support (3-4). Then run the test again.

This is most likely what we'll do when we get time to look into this
new test

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

11

Well, hopefully the situation will be better in the near future. As I said, the OGC is restructuring the governance of CITE.

Actually in my first post I was asking about disabling some SRS: I've tried to do that, but can't really figure out how...
What is the "WMS configuration" you refer to?

Thanks,
  L

Il giorno 31/mag/2010, alle ore 15.15, Andrea Aime ha scritto:

Andrea Aime ha scritto:

I interrupted the test after some time.
I reckon you are running the tests from localhost, nevertheless I would doubt that so many tests (and many more others before and after those in the picture) can be executed in one minute.

Btw, there is a simple workaround for this one: go into the
WMS configuration and declare a limited list of EPSG codes you
want to support (3-4). Then run the test again.

This is most likely what we'll do when we get time to look into this
new test

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

---
Dott. Lorenzo Bigagli
Istituto di Metodologie per l'Analisi Ambientale
del Consiglio Nazionale delle Ricerche (IMAA-CNR)

i: Area della Ricerca di Potenza, Contrada Santa Loja
   Zona Industriale, 85050 Tito Scalo (PZ), Italia
t: +39 0971 427221
f: +39 0971 427222
m: lorenzo.bigagli@anonymised.com

12

Andrea Aime ha scritto:

Andrea Aime ha scritto:

I haven't been involved in CITE testing for quite a bit of time
so no, I'm not sure. Justin should be able to provide more information,
he automated the CITE test suites we're running each night only
some time ago.

Btw, just had a quick look. We're now running the CITE tests as they
come from the OGC private svn as of revision 2740 (one needs a OGC
portal username/password to access it).

The version control is now at revision 3331, though only 4 commits
happened on the wms 1.1 directory after 2740. Looking at the diff
I can see there are substantial changes (the diff, as obtained
with https://svn.opengeospatial.org:8443/ogc-projects/cite/scripts/wms/1.1.1/
is 1.2MB!!!).

The diff was actually obtained running:

svn diff https://svn.opengeospatial.org:8443/ogc-projects/cite/scripts/wms/1.1.1/ -r 2740:HEAD > /tmp/wms.diff

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

13

Lorenzo Bigagli ha scritto:

Well, hopefully the situation will be better in the near future. As I said, the OGC is restructuring the governance of CITE.

Finger crossed.

Actually in my first post I was asking about disabling some SRS: I've tried to do that, but can't really figure out how...
What is the "WMS configuration" you refer to?

I see this bit of information has still not been ported from the old documentation to the new one. Anyways, here is the reference to the
old docs:

http://geoserver.org/display/GEOSDOC/WMS+Configuration

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

14

Lorenzo Bigagli ha scritto:

Limiting the number of srs's the test came to an end (81 pass, 9 fail).

Interesting, do you have any information about those 9 failures?
Sometimes failure are actually related to a misconfiguration (e.g.,
I remember getting CITE failures in some suite because
I exchanged the low/hight update sequence values)

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

15

I haven't had time to dig into it.
I attach the log files.

Regards,
L

s0003.zip (570 KB)

16

Sorry for chiming in late on this one. A few notes:

* Indeed I did limit the SRS list in the citewms-1.1.1 configuration that is running on the build server, the reason being that the recent changes to the wms cite tests added a test that looped over every single srs declared in the capabilities document. I will commit this change back into svn if there are no objections.

* To get the latest version of the wms cite tests required a lot of hacking of the tests themselves. I am trying to get those fixes committed back in but the process is not really moving anywhere.

On 10-05-31 7:50 AM, Andrea Aime wrote:

Lorenzo Bigagli ha scritto:

Limiting the number of srs's the test came to an end (81 pass, 9 fail).

Interesting, do you have any information about those 9 failures?
Sometimes failure are actually related to a misconfiguration (e.g.,
I remember getting CITE failures in some suite because
I exchanged the low/hight update sequence values)

Cheers
Andrea

--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.

17

Justin Deoliveira ha scritto:

Sorry for chiming in late on this one. A few notes:

* Indeed I did limit the SRS list in the citewms-1.1.1 configuration that is running on the build server, the reason being that the recent changes to the wms cite tests added a test that looped over every single srs declared in the capabilities document. I will commit this change back into svn if there are no objections.

Absolutely not, go ahead :slight_smile:

* To get the latest version of the wms cite tests required a lot of hacking of the tests themselves. I am trying to get those fixes committed back in but the process is not really moving anywhere.

Ok, so you identified those failures as bugs in the cite test suite?
That wouldn't be news, CITE tests are quite buggy for being an
official certification test, as everyone that ever tried to pass
it could attest :wink:

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

18

On 31/05/10 21:20, Lorenzo Bigagli wrote:

Well, hopefully the situation will be better in the near future. As I said, the OGC is restructuring the governance of CITE.

Lorenzo, please tell us more.

Are you involved in this restructuring? Is Raj Singh (OGC Interoperability Programs Director) involved? Do you know what it will entail?

I have two issues on my agenda:

(1) Improved (i.e. open) read-access to OGC resources (standards, drafts, and data). This includes CITE resources.

(2) We (AuScope/SISS/ASRDC) are developing a Reference Data Set for complex features WFS. I would like this to become a CITE test suite.

I have not received a reply from Raj.

Kind regards,

--
Ben Caradoc-Davies <Ben.Caradoc-Davies@anonymised.com>
Software Engineering Team Leader
CSIRO Earth Science and Resource Engineering
Australian Resources Research Centre

19

OGC is seeking to hire a director of the interoperability certification, with overall responsibility for OGC’s interoperability compliance development.

We have also proposed some possible contributions to the CSW test suite: I guess the procedure is just slow, for the time being.
Regards,
  L

Il giorno 01/giu/2010, alle ore 05.17, Ben Caradoc-Davies ha scritto:

On 31/05/10 21:20, Lorenzo Bigagli wrote:

Well, hopefully the situation will be better in the near future. As I said, the OGC is restructuring the governance of CITE.

Lorenzo, please tell us more.

Are you involved in this restructuring? Is Raj Singh (OGC Interoperability Programs Director) involved? Do you know what it will entail?

I have two issues on my agenda:

(1) Improved (i.e. open) read-access to OGC resources (standards, drafts, and data). This includes CITE resources.

(2) We (AuScope/SISS/ASRDC) are developing a Reference Data Set for complex features WFS. I would like this to become a CITE test suite.

I have not received a reply from Raj.

Kind regards,

--
Ben Caradoc-Davies <Ben.Caradoc-Davies@anonymised.com>
Software Engineering Team Leader
CSIRO Earth Science and Resource Engineering
Australian Resources Research Centre

---
Dott. Lorenzo Bigagli
Istituto di Metodologie per l'Analisi Ambientale
del Consiglio Nazionale delle Ricerche (IMAA-CNR)

i: Area della Ricerca di Potenza, Contrada Santa Loja
   Zona Industriale, 85050 Tito Scalo (PZ), Italia
t: +39 0971 427221
f: +39 0971 427222
m: lorenzo.bigagli@anonymised.com