[Geoserver-devel] New CITE test tooling ready for the brave

Hi,
as you probably know, I’ve lately been working on upgrading the nightly CITE tests tooling,
preparing it for an upgrade.

The current result of the work is in this branch:
https://github.com/geoserver/geoserver-cite-tools/tree/te-2015

Just like the current version, the TeamEngine and all the tests we
care for are imported as submodule. Unlike the current version,
the submodules import directly from specific revisions of the official
repositories, the idea is that we’re going to try to track the
official tests, and if we need a fix, we are going to make pull
requests against the official repos.

The README.md has been updated with instructions on how
to update a test, or add a new one, and has a section at the
end on how one is supposed to integrate with a build server.

The direct integration with the external repos, and the fact that
scripts are now build with maven, made the overall build
that much slower compared to the old approach (where the
tests were just a bunch of static files), so for the build
server integration I’ve split the scripts in two, one that builds
all the tooling, and one to run a specific tests afterwards.

For manual usage I suggest to just use the webapp instead,
with “ant webapp”.

So, in short, if you want to try the current tests on your machine,
I would suggest to:

  • Checkout the tools
  • Switch to the te-2015 branch
  • For the first time, run “git submodule update --init” in order
    to get all the submodule
  • ant clean build
  • ant webapp

Then connect at “http://localhost:9090/teamengine” and start
running the CITE tests you want.

The scripts available are the following ones. I did run all of them
at one point, but for most, I don’t remember if they pass or not:

  • ets-csw202: used to pass, does not anymore
  • ets-wcs10: don’t remember
  • ets-wcs11: don’t remember
  • ets-wfs10: not sure but I believe it passed
  • ets-wfs11: does not pass anymore
  • ets-wfs20: new, and problematic, not part of the build yet,
    makes a classpath mess with the other modules. Also strange,
    it does not require specific test data, we might want to hack the
    cite.sh script a bit and just use the wfs 1.1 data directory instead of
    creating a specific one for it. If you are running the webapp, just
    load the wfs 1.1 data directory
  • ets-wms11: not sure but I believe it passed
  • ets-wms13: not sure but I believe it passed
  • ets-wmts10: new
  • ets-wps10: new

From the above, we’re also missing ets-wcs20 which I still haven’t set up.

If we want to ugprade to the new CITE tests it’s now time to swarm
this. We need developers to take on a specific suite and start fixing
failures, as well as someone trying to setup these new tests on
Hudson.

Everything is pretty green, so expect issues from either the setup
or the tests themselves, and of course from GeoServer itself.

So, do we have any takers? I’m pretty tired by now, really need some
help to get this going :slight_smile:

Cheers
Andrea

PS: for the future, there are also the GML and KML compliance tests that we might
eventually want to run, by starting a GeoServer, dumping the files
from WFS/WMS and pass them to the tests.
A WMS client test could also be interesting, if we can script some
interactions between our GeoTools WMS client library and a running
GeoServer.

···

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


On Sat, 1 Aug 2015 06:00:09 PM Andrea Aime wrote:

The current result of the work is in this branch:
https://github.com/geoserver/geoserver-cite-tools/tree/te-2015

I cloned it, and had a few problems. I eventually got it going though.
Just one PR for OGC code, and one PR for our code so far. One more
PR is likely needed: see below.

Do I need Jira tickets for geoserver-cite-tools fixes? If so, should they
be filed against CITE component in geoserver?

Just like the current version, the TeamEngine and all the tests we
care for are imported as submodule. Unlike the current version,
the submodules import directly from specific revisions of the official
repositories, the idea is that we're going to try to track the
official tests, and if we need a fix, we are going to make pull
requests against the official repos.

So if there is a problem with the OGC module, we fix it there? Not try to work
around it?

Example: I couldn't get the ant task to build. Turned out WFS 1.1
(and WCS 1.0) builds fail on my Linux box with JDK 8 (even if I run the
mvn task directly in the checked-out code). I decided:

diff --git a/build.xml b/build.xml
index dc8c31e..6111afe 100644
--- a/build.xml
+++ b/build.xml
@@ -112,7 +112,7 @@
       <arg line="install -DskipTests -nsu" />
     </exec>
     <exec dir="${basedir}/ets-wfs11" executable="mvn">
- <arg line="install -DskipTests -nsu" />
+ <arg line="install -DskipTests -nsu -Dmaven.compiler.source=1.5 -Dmaven.compiler.target=1.5" />
     </exec>
     <exec dir="${basedir}/ets-wcs10" executable="mvn">
       <arg line="install -DskipTests -nsu" />

however specifying a source and target version inside the WFS 1.1 pom.xml
would also be an option. Which is preferred?

So, in short, if you want to try the current tests on your machine,
I would suggest to:
* Checkout the tools
* Switch to the te-2015 branch
* For the first time, run "git submodule update --init" in order
   to get all the submodule
* ant clean build
* ant webapp

Then connect at "http://localhost:9090/teamengine&quot; and start
running the CITE tests you want.

Some of the tests also need data loaded, right? So should that
go into README.md?

- ets-wms11: not sure but I believe it passed

I started here, and its failing. It looks like at least some real bugs.
Should the approach be to log those issues into Jira in some
kind of logical group(s)? Then submit PRs as normal?

Brad

On Sun, Aug 2, 2015 at 5:44 AM, Brad Hards <bradh@anonymised.com> wrote:

On Sat, 1 Aug 2015 06:00:09 PM Andrea Aime wrote:
> The current result of the work is in this branch:
> https://github.com/geoserver/geoserver-cite-tools/tree/te-2015
I cloned it, and had a few problems. I eventually got it going though.
Just one PR for OGC code, and one PR for our code so far. One more
PR is likely needed: see below.

Do I need Jira tickets for geoserver-cite-tools fixes? If so, should they
be filed against CITE component in geoserver?

Hum... unsure, I did not think about it :slight_smile:
We probably don't want geoserver-cite-tools stuff to pollute the changelogs,
but if needed we can have the tickets in there, and not assign a
version number to them.

So, as you prefer, just don't assign a "fix for" version.

> Just like the current version, the TeamEngine and all the tests we
> care for are imported as submodule. Unlike the current version,
> the submodules import directly from specific revisions of the official
> repositories, the idea is that we're going to try to track the
> official tests, and if we need a fix, we are going to make pull
> requests against the official repos.
So if there is a problem with the OGC module, we fix it there? Not try to
work
around it?

Example: I couldn't get the ant task to build. Turned out WFS 1.1
(and WCS 1.0) builds fail on my Linux box with JDK 8 (even if I run the
mvn task directly in the checked-out code). I decided:

diff --git a/build.xml b/build.xml
index dc8c31e..6111afe 100644
--- a/build.xml
+++ b/build.xml
@@ -112,7 +112,7 @@
       <arg line="install -DskipTests -nsu" />
     </exec>
     <exec dir="${basedir}/ets-wfs11" executable="mvn">
- <arg line="install -DskipTests -nsu" />
+ <arg line="install -DskipTests -nsu -Dmaven.compiler.source=1.5
-Dmaven.compiler.target=1.5" />
     </exec>
     <exec dir="${basedir}/ets-wcs10" executable="mvn">
       <arg line="install -DskipTests -nsu" />

however specifying a source and target version inside the WFS 1.1 pom.xml
would also be an option. Which is preferred?

I would do both, the one you made now to get you going, but
also push a change to the official repos if possible, e..g, in the long
run I believe it's best if we have the least amount of work arounds,
but we also have to avoid getting stuck in the short term.

> So, in short, if you want to try the current tests on your machine,
> I would suggest to:
> * Checkout the tools
> * Switch to the te-2015 branch
> * For the first time, run "git submodule update --init" in order
> to get all the submodule
> * ant clean build
> * ant webapp
>
> Then connect at "http://localhost:9090/teamengine&quot; and start
> running the CITE tests you want.
Some of the tests also need data loaded, right? So should that
go into README.md?

> - ets-wms11: not sure but I believe it passed
I started here, and its failing. It looks like at least some real bugs.
Should the approach be to log those issues into Jira in some
kind of logical group(s)? Then submit PRs as normal?

Yeah, if we have issues in the GeoServer code, business as usual
I'd say.
About WMS, one thing, also the old engine is failing, which may
be due to our yesterday's merge of the "remote execution vulnerability"
fixes affecting our xstream deserialization. Just saying, there is a small
chance it might be configuration related:
http://ares.boundlessgeo.com/jenkins/view/geoserver-cite/job/cite-wms-1.1/660/console

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------