[Geoserver-devel] [Geoserver-users] netcdf in non latlon grid

Hi,

Because no response on user list, I'm forwarding my (too long?) question
to here hoping to meet someone with deeper netcdf implementation knowledge.

tldr; Should it be able to server a netcdf with data with elevation AND
time in a normal/known epsg crs (non latlon, but like a national epsg).
I fail to load it in Geoserver. A 'simple' netcdf in epsg:xxxx without
time or elevation loads fine.

Regards,

Richard Duivenvoorde

PS Example data + python to create it here: http://duif.net/ncgeoserver.zip

-------- Forwarded Message --------
Subject: [Geoserver-users] netcdf in non latlon grid
Date: Wed, 9 Jan 2019 18:15:20 +0100
From: Richard Duivenvoorde <rdmailings@anonymised.com>
Reply-To: richard@anonymised.com
To: GeoServer Mailing List List <geoserver-users@lists.sourceforge.net>

Hi List,

I've succesfully created a simple minimal netcdf file (using Python)
with time-range-based data in a latlon grid:
like: float fakedata(time=3, z=3, lat=3, lon=4)

I can load the file (with several layers) both in QGIS (as mesh) and in
Geoserver (2.14.1 with netcdf extension) (with both time and elevation
dimensions): NICE!

BUT: we also want to project the data to our national grid: epsg:28992

If I understand docs correct, this should be possible (so: a known epsg,
no custom crs)?

Using gdal somebody provided me with a simple example in which instead
of lat lon the coordinates are x and y, and a CRS definition is defined:
"oblique_stereographic" and used as 'grid_mapping'.
But this is only 2D: (y=3, x=4)

With my python-netcdf-creation script I'm able to do that, and Geoserver
loads the data as one layer.
But trying to add time and/or z I fail:

Caused by: org.geotools.feature.IllegalAttributeException:Unknown
attribute time:null value:null
  at
org.geotools.feature.simple.SimpleFeatureImpl.setAttribute(SimpleFeatureImpl.java:238)
  at
org.geotools.imageio.netcdf.VariableAdapter.setFeatureTime(VariableAdapter.java:1327)
  at
org.geotools.imageio.netcdf.VariableAdapter.createFeature(VariableAdapter.java:1280)
  at
org.geotools.imageio.netcdf.VariableAdapter.getFeatures(VariableAdapter.java:1232)
  at
org.geotools.imageio.netcdf.NetCDFImageReader.initIndex(NetCDFImageReader.java:363)

or other silent issues.

I tried to google and look into the code to find the magic combination
of attributes/variables needed to be defined, but after hours of
fiddling, I tend to give up and ask for help :slight_smile:

So Question: should Geoserver be able to serve
fakedata(time=3, z=3, y=3, x=4)
with given crs, just as it is able to serve:
fakedata(time=3, z=3, lat=3, lon=4) ?

If anybody has a working example of such a grid I'm also happy, as that
proves that my feeling is right that it should be possible and I miss
something...

The python code and working latlon and not-working epsg28992 output
netcdf's are here http://duif.net/ncgeoserver.zip if somebody is willing
to have a look.

Regards,

Richard Duivenvoorde

_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to
this list:
- Earning your support instead of buying it, but Ian Turton:
http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines:
http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this:
https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer

Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

Richard,
if you get no answer on the user list, it means that nobody knows the answer, has time to answer,
or has time to investigate. Please use the devel list only if you are participating to the actual development.

Thanks for your understanding.

Cheers
Andrea

···

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

On 1/11/19 10:00 AM, Andrea Aime wrote:

Richard,
if you get no answer on the user list, it means that nobody knows the
answer, has time to answer,
or has time to investigate. Please use the devel list only if you are
participating to the actual development.

Ok, sorry.

It's just that I've invested some days already to get it going, and

https://docs.geoserver.org/stable/en/user/extensions/netcdf/netcdf.html#supporting-custom-netcdf-coordinate-reference-systems

gives me the feeling it should (and I just fail to call the magic
spells). I was hoping a dev with implementation knowledge could confirm
it SHOULD work. A 'NYI" would do too...

Regards,

Richard

Hi Richard.
Today I have downloaded GeoServer 2.14.1, together with NetCDF extension and I have configured your sample data.
I have simply created a NetCDF store, pointing to your test4dz28992.nc file.
I was able to configure the layer and related dimensions. I have setup a fake style so you can see the 4 different values as 4 different colors.

This is the result of the openLayers preview (getMap request with default values).

fakedata-default.jpg

Next one, is same getMap request with elevation=100.

fakedata-elevation100.jpg

So, I’m not sure why you are having issues in configuring it.
Could you retry from a clean situation?
Wondering if you successfully configured the dataset without t/z and then you replaced the file without reconfiguring it or something similar so that the configuration doesn’t reflect the new data.

As a quick check, you may want to move the sample file to another location and reconfigure it, in order to make sure it isn’t using any preconfigured/partially configured ancillary file.
Are you configuring it as a simple NetCDF store? (I mean… not an ImageMosaic)

Please, let us know.
Cheers,
Daniele

···

Regards,
Daniele Romagnoli

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it


Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

On 1/14/19 12:23 PM, Daniele Romagnoli wrote:

Hi Richard.
Today I have downloaded GeoServer 2.14.1, together with NetCDF extension
and I have configured your sample data.
I have simply created a NetCDF store, pointing to your test4dz28992.nc
<http://test4dz28992.nc> file.
I was able to configure the layer and related dimensions. I have setup a
fake style so you can see the 4 different values as 4 different colors.

This is the result of the openLayers preview (getMap request with
default values).
fakedata-default.jpg
Next one, is same getMap request with elevation=100.
fakedata-elevation100.jpg

So, I'm not sure why you are having issues in configuring it.
Could you retry from a clean situation?
Wondering if you successfully configured the dataset without t/z and
then you replaced the file without reconfiguring it or something similar
so that the configuration doesn't reflect the new data.
As a quick check, you may want to move the sample file to another
location and reconfigure it, in order to make sure it isn't using any
preconfigured/partially configured ancillary file.
Are you configuring it as a simple NetCDF store? (I mean... not an
ImageMosaic)

Hi Daniele,

THANKS! For trying and testing, and letting me know it was working,
you've made my day ( see below :slight_smile: )!

I've already invested a lot of hours to try all kind of different
netcdf-configs creations with attributes/variables etc etc, and 2d was
working but 3d or 4d (that is with t/z) was NOT. I just was not sure if
my self created netcdf's were crippled, or I hit a geoserver problem.

I just downloaded a fresh 2.14.1 + netcdf extension, but had the same
error again...

BUT... because you are showing results, I figured out the netcdf is ok,
so it has to do with something else...
So I moved the test4dz28992.nc from my /home/richard/dev/python/netcdf
dir to the [geoserver]/datadir/coverages.. and THEN it works!!! Halleluya

So the crux is: while latlon-netcdfs work outside the datadir,
NON-latlon-netcdfs fail outside the datadir (though you can browse in
the store creation page to you home dir....).

Worth an issue I think?

Regards & thanks again, now I can proceed with my life \o/

Richard

Hi Richard
any chance that you are maybe using a different user running GeoServer which doesn’t have write access to that /home/richard folder (and subfolders)?
The NetCDF format creates/writes a couple of ancillary files to index the ND variables stored on the nc file. There might be the case that it was unable to create them due to permissions issues.

Daniele

···

Regards,
Daniele Romagnoli

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it


Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

On 1/14/19 2:39 PM, Daniele Romagnoli wrote:

Hi Richard
any chance that you are maybe using a different user running GeoServer
which doesn't have write access to that /home/richard folder (and
subfolders)?
The NetCDF format creates/writes a couple of ancillary files to index
the ND variables stored on the nc file. There might be the case that it
was unable to create them due to permissions issues.

Nope permissions should not be a problem, it's all done in my home dir,
both the writing/creation of the netcdf (via python), as running (as
user richard) the standalone Geoserver bin 2.14.1 etc

But did you test to put the nc file outside of the datadir? Like put it
in /tmp and 'browse' to it when you define the datastore?
With me that fails with a big red block of errors there (IF I do not
have the nc in the datadir/coverages)?

I could try to do this in a windows virt machine at home...

Regards,

Richard Duivenvoorde

Hi Richard
any chance that you are maybe using a different user running GeoServer
which doesn’t have write access to that /home/richard folder (and
subfolders)?
The NetCDF format creates/writes a couple of ancillary files to index
the ND variables stored on the nc file. There might be the case that it
was unable to create them due to permissions issues.

Nope permissions should not be a problem, it’s all done in my home dir,
both the writing/creation of the netcdf (via python), as running (as
user richard) the standalone Geoserver bin 2.14.1 etc

But did you test to put the nc file outside of the datadir? Like put it
in /tmp and ‘browse’ to it when you define the datastore?
With me that fails with a big red block of errors there (IF I do not
have the nc in the datadir/coverages)?

Mmm. I only did the test on my Windows workstation. I didn’t test on Linux yet.
sorry…

Daniele

···

Regards,
Daniele Romagnoli

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it


Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia.

This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.

On 1/14/19 5:08 PM, Daniele Romagnoli wrote:

Mmm. I only did the test on my Windows workstation. I didn't test on
Linux yet.

Just installed 2.14.1+netcdf on a pretty fresh Win10 machine, and
confirmed my earlier finding: outside of the data directory creation of
a 4D nonlatlon netcdf store just fails. When you put exact the same nc
file inside the data dir, geoserver succeeds to create and serve the data.

Created an issue for it:
https://osgeo-org.atlassian.net/projects/GEOS/issues/GEOS-9098

Regards,

Richard Duivenvoorde

ps I'm not able to start 2.15-M0 with the netcdf extension (GEOS-9099)