Hi
This affects CSV output format for 2.19.2
The setting of ‘0’ in ‘Number of decimals (GML and GeoJSON output)’ for the Global settings, which for these output formats leaves the number of decimals unbounded, has the unfortunate result of truncating floats in CSV output to integers.
I have not checked if this affects other output formats. Is this bug already known or should I create a JIRA ticket?
Thanks
Michael Sexton
Data Analyst | MEGIS
Minerals Energy Groundwater | GEOSCIENCE AUSTRALIA
I would not think this is a bug. If there are zero decimal places how can any software reading the CSV know the number is meant to be a float?
This is a limitation of converting to a text file
From: Michael Sexton Michael.Sexton@anonymised.com
Sent: Tuesday, 2 November 2021 11:19 AM
To: ‘geoserver-users@lists.sourceforge.net’ geoserver-users@anonymised.coms.sourceforge.net
Subject: [Geoserver-users] ‘Zero’ for Number of decimals (GML and GeoJSON output) in Global settings truncates CSV floats to ints [SEC=UNOFFICIAL]
Hi
This affects CSV output format for 2.19.2
The setting of ‘0’ in ‘Number of decimals (GML and GeoJSON output)’ for the Global settings, which for these output formats leaves the number of decimals unbounded, has the unfortunate result of truncating floats in CSV output to integers.
I have not checked if this affects other output formats. Is this bug already known or should I create a JIRA ticket?
Thanks
Michael Sexton
Data Analyst | MEGIS
Minerals Energy Groundwater | GEOSCIENCE AUSTRALIA
__________________________________________________****__________
Phone: +61 2 6249 9262 Fax: +61 2 6249 9999
Email: Michael.Sexton@anonymised.com Web: www.ga.gov.au
Cnr Jerrabomberra Avenue and Hindmarsh Drive Symonston ACT 2609
GPO Box 378 Canberra ACT 2601 Australia
Applying geoscience to Australia’s most important challenges
Geoscience Australia Disclaimer: This e-mail (and files transmitted with it) is intended only for the person or entity to which it is addressed. If you are not the intended recipient, then you have received this e-mail by mistake and any use, dissemination, forwarding, printing or copying of this e-mail and its file attachments is prohibited. The security of emails transmitted cannot be guaranteed; by forwarding or replying to this email, you acknowledge and accept these risks.
Hi Graham
The problem is the setting has an inconsistent behaviour depending on output format
GML32 with number of decimal places 0:
<coit:latitude_gda94>
-18.8601667
</coit:latitude_gda94>
<coit:longitude_gda94>
138.6970111
</coit:longitude_gda94>
GeoJSON with number of decimal places 0:
“latitude_gda94”: -18.8601667,
“longitude_gda94”: 138.6970111,
CSV with number of decimal places 0:
latitude_gda94 |
longitude_gda94 |
|---|---|
-19 |
139 |
From: Humphries, Graham Graham.Humphries@anonymised.com
Sent: Tuesday, 2 November 2021 11:40 AM
To: Michael Sexton Michael.Sexton@anonymised.com; ‘geoserver-users@lists.sourceforge.net’ geoserver-users@lists.sourceforge.net
Subject: RE: ‘Zero’ for Number of decimals (GML and GeoJSON output) in Global settings truncates CSV floats to ints [SEC=UNOFFICIAL]
I would not think this is a bug. If there are zero decimal places how can any software reading the CSV know the number is meant to be a float?
This is a limitation of converting to a text file
From: Michael Sexton <Michael.Sexton@anonymised.com>
Sent: Tuesday, 2 November 2021 11:19 AM
To: ‘geoserver-users@lists.sourceforge.net’ <geoserver-users@lists.sourceforge.net>
Subject: [Geoserver-users] ‘Zero’ for Number of decimals (GML and GeoJSON output) in Global settings truncates CSV floats to ints [SEC=UNOFFICIAL]
Hi
This affects CSV output format for 2.19.2
The setting of ‘0’ in ‘Number of decimals (GML and GeoJSON output)’ for the Global settings, which for these output formats leaves the number of decimals unbounded, has the unfortunate result of truncating floats in CSV output to integers.
I have not checked if this affects other output formats. Is this bug already known or should I create a JIRA ticket?
Thanks
Michael Sexton
Data Analyst | MEGIS
Minerals Energy Groundwater | GEOSCIENCE AUSTRALIA
__________________________________________________****__________
Phone: +61 2 6249 9262 Fax: +61 2 6249 9999
Email: Michael.Sexton@anonymised.com Web: www.ga.gov.au
Cnr Jerrabomberra Avenue and Hindmarsh Drive Symonston ACT 2609
GPO Box 378 Canberra ACT 2601 Australia
Applying geoscience to Australia’s most important challenges
Geoscience Australia Disclaimer: This e-mail (and files transmitted with it) is intended only for the person or entity to which it is addressed. If you are not the intended recipient, then you have received this e-mail by mistake and any use, dissemination, forwarding, printing or copying of this e-mail and its file attachments is prohibited. The security of emails transmitted cannot be guaranteed; by forwarding or replying to this email, you acknowledge and accept these risks.
CONFIDENTIALITY NOTICE AND DISCLAIMER
The information in this transmission may be confidential and/or protected by legal professional privilege, and is intended only for the person or persons to whom it is addressed. If you are not such a person, you are warned that any disclosure, copying or dissemination of the information is unauthorised. If you have received the transmission in error, please immediately contact this office by telephone, fax or email, to inform us of the error and to enable arrangements to be made for the destruction of the transmission, or its return at our cost. No liability is accepted for any unauthorised use of the information contained in this transmission.
Another issue, GeoJSON coordinates exhibit the same truncation, even though feature properties don’t.
·
geometry": {
o “type”: “Point”,
o “coordinates”: [
§ 139,
§ -19,
§ 212
]
· },
· “geometry_name”: “geometry”,
· “properties”: {
o “eno”: 5757,
o “sampling_feature”: “DDH 83-1”,
o “sampling_feature_type”: “borehole”,
o “sampling_feature_subtype”: “petroleum - stratigraphic investigation”,
o “latitude_gda94”: -18.8601667,
o “longitude_gda94”: 138.6970111,
From: Michael Sexton Michael.Sexton@anonymised.com
Sent: Tuesday, 2 November 2021 12:17 PM
To: Humphries, Graham Graham.Humphries@anonymised.com; ‘geoserver-users@lists.sourceforge.net’ geoserver-users@lists.sourceforge.net
Subject: Re: [Geoserver-users] ‘Zero’ for Number of decimals (GML and GeoJSON output) in Global settings truncates CSV floats to ints [SEC=UNOFFICIAL]
Hi Graham
The problem is the setting has an inconsistent behaviour depending on output format
GML32 with number of decimal places 0:
<coit:latitude_gda94>
-18.8601667
</coit:latitude_gda94>
<coit:longitude_gda94>
138.6970111
</coit:longitude_gda94>
GeoJSON with number of decimal places 0:
“latitude_gda94”: -18.8601667,
“longitude_gda94”: 138.6970111,
CSV with number of decimal places 0:
latitude_gda94 |
longitude_gda94 |
|---|---|
-19 |
139 |
From: Humphries, Graham <Graham.Humphries@anonymised.com>
Sent: Tuesday, 2 November 2021 11:40 AM
To: Michael Sexton <Michael.Sexton@anonymised.com>; ‘geoserver-users@lists.sourceforge.net’ <geoserver-users@lists.sourceforge.net>
Subject: RE: ‘Zero’ for Number of decimals (GML and GeoJSON output) in Global settings truncates CSV floats to ints [SEC=UNOFFICIAL]
I would not think this is a bug. If there are zero decimal places how can any software reading the CSV know the number is meant to be a float?
This is a limitation of converting to a text file
From: Michael Sexton <Michael.Sexton@anonymised.com>
Sent: Tuesday, 2 November 2021 11:19 AM
To: ‘geoserver-users@lists.sourceforge.net’ <geoserver-users@lists.sourceforge.net>
Subject: [Geoserver-users] ‘Zero’ for Number of decimals (GML and GeoJSON output) in Global settings truncates CSV floats to ints [SEC=UNOFFICIAL]
Hi
This affects CSV output format for 2.19.2
The setting of ‘0’ in ‘Number of decimals (GML and GeoJSON output)’ for the Global settings, which for these output formats leaves the number of decimals unbounded, has the unfortunate result of truncating floats in CSV output to integers.
I have not checked if this affects other output formats. Is this bug already known or should I create a JIRA ticket?
Thanks
Michael Sexton
Data Analyst | MEGIS
Minerals Energy Groundwater | GEOSCIENCE AUSTRALIA
__________________________________________________****__________
Phone: +61 2 6249 9262 Fax: +61 2 6249 9999
Email: Michael.Sexton@anonymised.com Web: www.ga.gov.au
Cnr Jerrabomberra Avenue and Hindmarsh Drive Symonston ACT 2609
GPO Box 378 Canberra ACT 2601 Australia
Applying geoscience to Australia’s most important challenges
Geoscience Australia Disclaimer: This e-mail (and files transmitted with it) is intended only for the person or entity to which it is addressed. If you are not the intended recipient, then you have received this e-mail by mistake and any use, dissemination, forwarding, printing or copying of this e-mail and its file attachments is prohibited. The security of emails transmitted cannot be guaranteed; by forwarding or replying to this email, you acknowledge and accept these risks.
CONFIDENTIALITY NOTICE AND DISCLAIMER
The information in this transmission may be confidential and/or protected by legal professional privilege, and is intended only for the person or persons to whom it is addressed. If you are not such a person, you are warned that any disclosure, copying or dissemination of the information is unauthorised. If you have received the transmission in error, please immediately contact this office by telephone, fax or email, to inform us of the error and to enable arrangements to be made for the destruction of the transmission, or its return at our cost. No liability is accepted for any unauthorised use of the information contained in this transmission.
Geoscience Australia Disclaimer: This e-mail (and files transmitted with it) is intended only for the person or entity to which it is addressed. If you are not the intended recipient, then you have received this e-mail by mistake and any use, dissemination, forwarding, printing or copying of this e-mail and its file attachments is prohibited. The security of emails transmitted cannot be guaranteed; by forwarding or replying to this email, you acknowledge and accept these risks.
On Tue, 2 Nov 2021 at 02:11, Michael Sexton <Michael.Sexton@anonymised.com4…> wrote:
Another issue, GeoJSON coordinates exhibit the same truncation, even though feature properties don’t.
·
geometry": {o “type”: “Point”,
o “coordinates”: [
§ 139,
§ -19,
§ 212
]
· },
· “geometry_name”: “geometry”,
· “properties”: {
o “eno”: 5757,
o “sampling_feature”: “DDH 83-1”,
o “sampling_feature_type”: “borehole”,
o “sampling_feature_subtype”: “petroleum - stratigraphic investigation”,
o “latitude_gda94”: -18.8601667,
o “longitude_gda94”: 138.6970111,
This is what I would expect for a setting of 0 decimal places - it should only affect the coordinates and not the other parameters, but in a csv file there is no distinction between the coordinates and the properties which is a bug, Please open it against the GeoTools CSVDatastore and I’ll try to take a look over the weekend.
Ian
Hi Ian
Just for clarification, it’s not an issue with the datastore, but differing output formats for a WFS, ‘text/csv’ as an outputFormat truncates the values, GeoJSON/GML32 does not. The datastore in question is Postgres, but I don’t think that is material to this problem.
Thanks
Michael
On Tue, 2 Nov 2021 at 02:11, Michael Sexton <Michael.Sexton@…174…> wrote:
Another issue, GeoJSON coordinates exhibit the same truncation, even though feature properties don’t.
·
geometry": {o “type”: “Point”,
o “coordinates”: [
§ 139,
§ -19,
§ 212
]
· },
· “geometry_name”: “geometry”,
· “properties”: {
o “eno”: 5757,
o “sampling_feature”: “DDH 83-1”,
o “sampling_feature_type”: “borehole”,
o “sampling_feature_subtype”: “petroleum - stratigraphic investigation”,
o “latitude_gda94”: -18.8601667,
o “longitude_gda94”: 138.6970111,
This is what I would expect for a setting of 0 decimal places - it should only affect the coordinates and not the other parameters, but in a csv file there is no distinction between the coordinates and the properties which is a bug, Please open it against the GeoTools CSVDatastore and I’ll try to take a look over the weekend.
Ian
Hi Ian
Just for clarification, it’s not an issue with the datastore, but differing output formats for a WFS, ‘text/csv’ as an outputFormat truncates the values, GeoJSON/GML32 does not. The datastore in question is Postgres, but I don’t think that is material to this problem.
I think that the CSV datastore is used to write the data out, as are the GeoJSON and GML stores for those formats.
Ian
Ian Turton
Think again ;-). The output formats for CSV and GeoJSON predate the respective stores by many years.
A switch to the store for CSV might be possible (careful about making sure it’s streaming, and it’s performing),
one for GeoJSON is going to be significantly harder, it’s extended in many places for specific needs.
Cheers
Andrea
GeoServer Professional Services from the experts!
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions Group
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928
https://www.geosolutionsgroup.com/
http://twitter.com/geosolutions_it
Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.
This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail
Hi All
JIRA task lodged
On Wed, Nov 3, 2021 at 9:14 AM Ian Turton <ijturton@…84…> wrote:
I think that the CSV datastore is used to write the data out, as are the GeoJSON and GML stores for those formats.
Think again ;-). The output formats for CSV and GeoJSON predate the respective stores by many years.
A switch to the store for CSV might be possible (careful about making sure it’s streaming, and it’s performing),
one for GeoJSON is going to be significantly harder, it’s extended in many places for specific needs.
Cheers
Andrea
==
GeoServer Professional Services from the experts!
Ing. Andrea Aime
@geowolf
Technical Lead
GeoSolutions Group
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 333 8128928
https://www.geosolutionsgroup.com/
http://twitter.com/geosolutions_it
Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.
This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail