[Geoserver-users] Styling Sld using strReplace to ignore accented characters

GeoServer GeoServer User
1

Hello all,

I need some help with the regex of the function “strReplace” because according to the docs http://docs.geoserver.org/stable/en/user/filter/function_reference.html i can use any regular expression from java, but when i use the expression “\p{IsM}+” is not working as on java, i made the tests using the same version of java runtime JDK7.

The example string: “ábÈitõ”
Pattern: \p{IsM}+
Replace: “”

Java response: “abEito”
But the geoserver Response still having the accents.

<ogc:Function name=“strReplace”>
ogc:PropertyNamePROPERTY_NAME</ogc:PropertyName>
ogc:Literal\p{IsM}+</ogc:Literal>
ogc:Literal/
ogc:Literaltrue</ogc:Literal>
</ogc:Function>

Thanks.

2

On Wed, Feb 11, 2015 at 9:47 PM, Danilo da Silveira Figueira <
danilomalzao@anonymised.com> wrote:

Hello all,

I need some help with the regex of the function "strReplace" because
according to the docs
http://docs.geoserver.org/stable/en/user/filter/function_reference.html i
can use any regular expression from java, but when i use the expression
"\p{IsM}+" is not working as on java, i made the tests using the same
version of java runtime JDK7.

The example string: "ábÈitõ"
Pattern: \p{IsM}+
Replace: ""

Java response: "abEito"
But the geoserver Response still having the accents.

<ogc:Function name="strReplace">
<ogc:PropertyName>PROPERTY_NAME</ogc:PropertyName>
<ogc:Literal>\p{IsM}+</ogc:Literal>
<ogc:Literal/>
<ogc:Literal>true</ogc:Literal>
</ogc:Function>

Hum... I don't know, the code definitely uses java regular expressions:
https://github.com/geotools/geotools/blob/master/modules/library/main/src/main/java/org/geotools/filter/function/FilterFunction_strReplace.java
which uses in turn:
https://github.com/geotools/geotools/blob/master/modules/library/main/src/main/java/org/geotools/filter/function/StaticGeometry.java#L553

It may well be that your non ASCII chars get lost travelling from the db to
this expession?

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

3

Hi Andrea thanks for your reply,

Well i am using files instead of DB but i guess that the enconding problem can be the same, i dont have any clue about how to solve this.
My guesses, if it is a enconding problem, there is a tool that can automatically correct the *.idb files to utf-8, or can i specify a encondig to sld files use for interpreting the records?

Ps: In case someone is interested the server address csr.ufmg.br/maps

···

2015-02-12 4:58 GMT-02:00 Andrea Aime <andrea.aime@anonymised.com>:

On Wed, Feb 11, 2015 at 9:47 PM, Danilo da Silveira Figueira <danilomalzao@anonymised.com> wrote:

Hello all,

I need some help with the regex of the function “strReplace” because according to the docs http://docs.geoserver.org/stable/en/user/filter/function_reference.html i can use any regular expression from java, but when i use the expression “\p{IsM}+” is not working as on java, i made the tests using the same version of java runtime JDK7.

The example string: “ábÈitõ”
Pattern: \p{IsM}+
Replace: “”

Java response: “abEito”
But the geoserver Response still having the accents.

<ogc:Function name=“strReplace”>
ogc:PropertyNamePROPERTY_NAME</ogc:PropertyName>
ogc:Literal\p{IsM}+</ogc:Literal>
ogc:Literal/
ogc:Literaltrue</ogc:Literal>
</ogc:Function>

Hum… I don’t know, the code definitely uses java regular expressions:
https://github.com/geotools/geotools/blob/master/modules/library/main/src/main/java/org/geotools/filter/function/FilterFunction_strReplace.java

which uses in turn:
https://github.com/geotools/geotools/blob/master/modules/library/main/src/main/java/org/geotools/filter/function/StaticGeometry.java#L553

It may well be that your non ASCII chars get lost travelling from the db to this expession?

Cheers
Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

4

On Thu, Feb 12, 2015 at 3:19 PM, Danilo da Silveira Figueira <
danilomalzao@anonymised.com> wrote:

Hi Andrea thanks for your reply,

Well i am using files instead of DB but i guess that the enconding problem
can be the same, i dont have any clue about how to solve this.
My guesses, if it is a enconding problem, there is a tool that can
automatically correct the *.idb files to utf-8, or can i specify a encondig
to sld files use for interpreting the records?

What's an idb file?

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

5

Sorry, i mean to change the Shapefile encondig to utf-8.

What i said about idb, is because the string names of the shapefile usually are contained in a “idb” file.

att

···

2015-02-12 12:27 GMT-02:00 Andrea Aime <andrea.aime@anonymised.com>:

On Thu, Feb 12, 2015 at 3:19 PM, Danilo da Silveira Figueira <danilomalzao@anonymised.com> wrote:

Hi Andrea thanks for your reply,

Well i am using files instead of DB but i guess that the enconding problem can be the same, i dont have any clue about how to solve this.
My guesses, if it is a enconding problem, there is a tool that can automatically correct the *.idb files to utf-8, or can i specify a encondig to sld files use for interpreting the records?

What’s an idb file?

Cheers

Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

6

Hi Andrea,

i have tried changing the encoding of the server but it does not solved the problem, i guess that the problem is really in geoserver.

att

···

2015-02-12 11:32 GMT-03:00 Danilo da Silveira Figueira <danilomalzao@anonymised.com>:

Sorry, i mean to change the Shapefile encondig to utf-8.

What i said about idb, is because the string names of the shapefile usually are contained in a “idb” file.

att

2015-02-12 12:27 GMT-02:00 Andrea Aime <andrea.aime@…1107…>:

On Thu, Feb 12, 2015 at 3:19 PM, Danilo da Silveira Figueira <danilomalzao@anonymised.com> wrote:

Hi Andrea thanks for your reply,

Well i am using files instead of DB but i guess that the enconding problem can be the same, i dont have any clue about how to solve this.
My guesses, if it is a enconding problem, there is a tool that can automatically correct the *.idb files to utf-8, or can i specify a encondig to sld files use for interpreting the records?

What’s an idb file?

Cheers

Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

7

On Thu, Feb 12, 2015 at 3:32 PM, Danilo da Silveira Figueira <
danilomalzao@anonymised.com> wrote:

Sorry, i mean to change the Shapefile encondig to utf-8.

If the dbf file is really in utf8, have you tried changing the charset in
the data store config dialog?
There should be a drop down there if memory serves me right (I don't have a
geoserver handy right now)

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

8

Well, after changing the storage encondig the strings are showed correctly in the WMS respose but the records yet still having accents. =/

···

2015-02-12 12:54 GMT-03:00 Andrea Aime <andrea.aime@anonymised.com7…>:

On Thu, Feb 12, 2015 at 3:32 PM, Danilo da Silveira Figueira <danilomalzao@anonymised.com> wrote:

Sorry, i mean to change the Shapefile encondig to utf-8.

If the dbf file is really in utf8, have you tried changing the charset in the data store config dialog?
There should be a drop down there if memory serves me right (I don’t have a geoserver handy right now)

Cheers

Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

9

It fails even when i try something like this:

áéíÀÈ`RÌ \p{IsM}+ a true
···

2015-02-12 13:32 GMT-03:00 Danilo da Silveira Figueira <danilomalzao@anonymised.com>:

Well, after changing the storage encondig the strings are showed correctly in the WMS respose but the records yet still having accents. =/

2015-02-12 12:54 GMT-03:00 Andrea Aime <andrea.aime@anonymised.com>:

On Thu, Feb 12, 2015 at 3:32 PM, Danilo da Silveira Figueira <danilomalzao@anonymised.com> wrote:

Sorry, i mean to change the Shapefile encondig to utf-8.

If the dbf file is really in utf8, have you tried changing the charset in the data store config dialog?
There should be a drop down there if memory serves me right (I don’t have a geoserver handy right now)

Cheers

Andrea

–

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/NWWaa2 for more information.

==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.