[Geoserver-devel] Master build is failing in restconfig due to a character encoding issue with rest resource tests

Hi,
we have the master build failing on OpenJDK8 in restconfig due to an apparent special char
management issue:

Tests run: 17, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.895 sec <<< FAILURE!
testSpecialCharacterNames(org.geoserver.rest.resources.ResourceTest) Time elapsed: 16 sec <<< FAILURE!
junit.framework.ComparisonFailure: expected:<…rce/po%C3%ABzie/caf%[C3%A9]> but was:<…rce/po%C3%ABzie/caf%[3F]>
at junit.framework.Assert.assertEquals(Assert.java:100)
at junit.framework.Assert.assertEquals(Assert.java:107)
at org.custommonkey.xmlunit.XMLAssert.assertXpathEvaluatesTo(XMLAssert.java:858)

http://build.geo-solutions.it/jenkins/job/GeoServer-Master-OpenJDK8/

I’ve tried locally and the test is not failing for me on openjdk 8.
Looking into it,it seems at least one char is “?” (%3F) so my guess it that the code should
have Assume checks verifying that the underlying file system can actually encode those
file names to start with

Cheers
Andrea

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


Hi,
investigated a bit more, the build fails reliably under the POSIX locale, e.g., if you export
LC_ALL=POSIX
before building.

The files built on the file system actually have the special chars replaced with ? in that case:

Inline image 1

The situation is a bit weird though in that trying to get the files with the actual name,
e.g., “/poëzie/café”, makes Java actually return them and claim the file name is the
one passed in the constructor, to get the ? one has to use list() against the directory
so that the code actually reads the names from the file system.
And this is exactly what REST does to represent the directory in XML, it ends up
calling list(), resulting in the directory being properly encoded (because of the direct access
by name) but the files not (because it was actually read using list() from the file system).

I ended up setting up a working Assume by doing the opposite, checking that po?zie
is not there. This makes the test run with UTF locales, but be skipped with POSIX ones:
https://github.com/geoserver/geoserver/pull/1621

Cheers
Andrea

···

On Tue, May 24, 2016 at 4:15 PM, Andrea Aime <andrea.aime@anonymised.com> wrote:

Hi,
we have the master build failing on OpenJDK8 in restconfig due to an apparent special char
management issue:

Tests run: 17, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.895 sec <<< FAILURE!
testSpecialCharacterNames(org.geoserver.rest.resources.ResourceTest) Time elapsed: 16 sec <<< FAILURE!
junit.framework.ComparisonFailure: expected:<…rce/po%C3%ABzie/caf%[C3%A9]> but was:<…rce/po%C3%ABzie/caf%[3F]>
at junit.framework.Assert.assertEquals(Assert.java:100)
at junit.framework.Assert.assertEquals(Assert.java:107)
at org.custommonkey.xmlunit.XMLAssert.assertXpathEvaluatesTo(XMLAssert.java:858)

http://build.geo-solutions.it/jenkins/job/GeoServer-Master-OpenJDK8/

I’ve tried locally and the test is not failing for me on openjdk 8.
Looking into it,it seems at least one char is “?” (%3F) so my guess it that the code should
have Assume checks verifying that the underlying file system can actually encode those
file names to start with

Cheers
Andrea

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313

fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


Strange... Thanks for fixing!

On 25-05-16 11:06, Andrea Aime wrote:

Hi,
investigated a bit more, the build fails reliably under the POSIX locale, e.g., if you export
LC_ALL=POSIX
before building.

The files built on the file system actually have the special chars replaced with ? in that case:

Inline image 1

The situation is a bit weird though in that trying to get the files with the actual name,
e.g., "/poëzie/café", makes Java actually return them and claim the file name is the
one passed in the constructor, to get the ? one has to use list() against the directory
so that the code actually reads the names from the file system.
And this is exactly what REST does to represent the directory in XML, it ends up
calling list(), resulting in the directory being properly encoded (because of the direct access
by name) but the files not (because it was actually read using list() from the file system).

I ended up setting up a working Assume by doing the opposite, checking that po?zie
is not there. This makes the test run with UTF locales, but be skipped with POSIX ones:
https://github.com/geoserver/geoserver/pull/1621

Cheers
Andrea

On Tue, May 24, 2016 at 4:15 PM, Andrea Aime <andrea.aime@anonymised.com <mailto:andrea.aime@anonymised.com>> wrote:

    Hi,
    we have the master build failing on OpenJDK8 in restconfig due to
    an apparent special char
    management issue:

     Tests run: 17, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
    0.895 sec <<< FAILURE!
    testSpecialCharacterNames(org.geoserver.rest.resources.ResourceTest)
     Time elapsed: 16 sec <<< FAILURE!
    junit.framework.ComparisonFailure:
    expected:<...rce/po%C3%ABzie/caf%[C3%A9]> but
    was:<...rce/po%C3%ABzie/caf%[3F]>
    at junit.framework.Assert.assertEquals(Assert.java:100)
    at junit.framework.Assert.assertEquals(Assert.java:107)
    at
    org.custommonkey.xmlunit.XMLAssert.assertXpathEvaluatesTo(XMLAssert.java:858)

    http://build.geo-solutions.it/jenkins/job/GeoServer-Master-OpenJDK8/

    I've tried locally and the test is not failing for me on openjdk 8.
    Looking into it,it seems at least one char is "?" (%3F) so my
    guess it that the code should
    have Assume checks verifying that the underlying file system can
    actually encode those
    file names to start with

    Cheers
    Andrea

    -- ==
    GeoServer Professional Services from the experts! Visit
    http://goo.gl/it488V for more information.
    ==

    Ing. Andrea Aime
    @geowolf
    Technical Lead

    GeoSolutions S.A.S.
    Via di Montramito 3/A
    55054 Massarosa (LU)
    phone: +39 0584 962313 <tel:%2B39%200584%20962313>
    fax: +39 0584 1660272 <tel:%2B39%200584%201660272>
    mob: +39 339 8844549 <tel:%2B39%20%C2%A0339%208844549>

    http://www.geo-solutions.it
    http://twitter.com/geosolutions_it

    *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

    Le informazioni contenute in questo messaggio di posta elettronica
    e/o nel/i file/s allegato/i sono da considerarsi strettamente
    riservate. Il loro utilizzo è consentito esclusivamente al
    destinatario del messaggio, per le finalità indicate nel messaggio
    stesso. Qualora riceviate questo messaggio senza esserne il
    destinatario, Vi preghiamo cortesemente di darcene notizia via
    e-mail e di procedere alla distruzione del messaggio stesso,
    cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
    divulgarlo anche in parte, distribuirlo ad altri soggetti,
    copiarlo, od utilizzarlo per finalità diverse, costituisce
    comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

    The information in this message and/or attachments, is intended
    solely for the attention and use of the named addressee(s) and may
    be confidential or proprietary in nature or covered by the
    provisions of privacy act (Legislative Decree June, 30 2003,
    no.196 - Italy's New Data Protection Code).Any use not in accord
    with its purpose, any disclosure, reproduction, copying,
    distribution, or either dissemination, either whole or partial, is
    strictly forbidden except previous formal approval of the named
    addressee(s). If you are not the intended recipient, please
    contact immediately the sender by telephone, fax or e-mail and
    delete the information in this message that has been received in
    error. The sender does not give any warranty or accept liability
    as the content, accuracy or completeness of sent messages and
    accepts no responsibility for changes made after they were sent or
    for other risks which arise as a result of e-mail transmission,
    viruses, etc.

    -------------------------------------------------------

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

-------------------------------------------------------