[Geoserver-users] Exposing NETCDF files as Image Mosaic accross a time index

GeoServer GeoServer User
1

Greetings,

I have been attempting to create an image mosaic for NETCDF files like this one (http://54.183.9.182/wff_153080300.nc) that I want to index by the time dimension in order to add additional forecast files as they become available. I am using the 2.9 SNAPSHOT version from 11/17 for all components.

I have tried to simply specify the directory within which this file sits as an image mosaic store, but this results in the following error :

19 Nov 01:01:54 WARN [netcdf.NetCDFGeoreferenceManager] - Unsupported axis: double lat(y=299, x=399);

:units = “degrees_north”;

:long_name = “latitude”;

:standard_name = “latitude”;

:_CoordinateAxisType = “Lat”;

in input: /gdata2/netcf/wff/wff_153080300.nc has been found

19 Nov 01:01:54 WARN [netcdf.NetCDFGeoreferenceManager] - Unsupported axis: double lon(y=299, x=399);

:units = “degrees_east”;

:long_name = “longitude”;

:standard_name = “longitude”;

:_CoordinateAxisType = “Lon”;

in input: /gdata2/netcf/wff/wff_153080300.nc has been found

19 Nov 01:03:28 ERROR [imagemosaic.Utils] -

java.lang.NullPointerException

19 Nov 01:03:28 WARN [gce.imagemosaic] - Failure occurred while collecting the granules

org.apache.commons.io.DirectoryWalker$CancelException: Operation Cancelled

at org.apache.commons.io.DirectoryWalker.checkIfCancelled(DirectoryWalker.java:395)

at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:368)

at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:335)

at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker$MosaicDirectoryWalker.(ImageMosaicDirectoryWalker.java:114)

at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker.run(ImageMosaicDirectoryWalker.java:196)

at org.geotools.gce.imagemosaic.Utils.createMosaic(Utils.java:360)

at org.geotools.gce.imagemosaic.Utils.checkSource(Utils.java:1250)

at org.geotools.gce.imagemosaic.ImageMosaicReader.initReaderFromURL(ImageMosaicReader.java:519)

at org.geotools.gce.imagemosaic.ImageMosaicReader.(ImageMosaicReader.java:471)

at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:483)

at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:103)

at org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1441)

at org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1369)

at org.geoserver.catalog.impl.CoverageStoreInfoImpl.getGridCoverageReader(CoverageStoreInfoImpl.java:59)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.geoserver.catalog.impl.ModificationProxy.invoke(ModificationProxy.java:146)

at com.sun.proxy.$Proxy53.getGridCoverageReader(Unknown Source)

at org.geoserver.web.data.layer.NewLayerPageProvider.getItemsInternal(NewLayerPageProvider.java:90)

at org.geoserver.web.data.layer.NewLayerPageProvider.getItems(NewLayerPageProvider.java:59)

at org.geoserver.web.wicket.GeoServerDataProvider.fullSize(GeoServerDataProvider.java:245)

I have tried to specify properties to the indexer in indexer.properties as outlined here (http://geoserver.geo-solutions.it/multidim/en/imagemosaic/index.html) but that produces the following error:

19 Nov 01:12:16 WARN [netcdf.NetCDFGeoreferenceManager] - Unsupported axis: double lat(y=299, x=399);

:units = “degrees_north”;

:long_name = “latitude”;

:standard_name = “latitude”;

:_CoordinateAxisType = “Lat”;

in input: /gdata2/netcf/wff/wff_153080300.nc has been found

19 Nov 01:12:16 WARN [netcdf.NetCDFGeoreferenceManager] - Unsupported axis: double lon(y=299, x=399);

:units = “degrees_east”;

:long_name = “longitude”;

:standard_name = “longitude”;

:_CoordinateAxisType = “Lon”;

in input: /gdata2/netcf/wff/wff_153080300.nc has been found

19 Nov 01:12:18 WARN [imagemosaic.catalog] - This granule catalog was not properly dispose as it still points to:ServiceInfo

description=Features from ShapefileDataStore

19 Nov 01:13:49 ERROR [imagemosaic.Utils] - Invalid mosaic schema SimpleFeatureTypeImpl cloud identified extends Feature(x:x,y:y,lat:lat,lon:lon,surface_temperature:surface_temperature,surface_dewpoint:surface_dewpoint,surface_rh:surface_rh,surface_wind_speed:surface_wind_speed,surface_wind_direction:surface_wind_direction,mixing_height:mixing_height,lifted_index:lifted_index,haines_middle:haines_middle,haines_high:haines_high,ventilation_index:ventilation_index,cloud:cloud,precipitation:precipitation,z:z,height:height,temperature:temperature,relative_humidity:relative_humidity,wind_speed:wind_speed,wind_direction:wind_direction,time:time), it should have a geometry and a location property of name location

java.lang.IllegalArgumentException: Invalid mosaic schema SimpleFeatureTypeImpl cloud identified extends Feature(x:x,y:y,lat:lat,lon:lon,surface_temperature:surface_temperature,surface_dewpoint:surface_dewpoint,surface_rh:surface_rh,surface_wind_speed:surface_wind_speed,surface_wind_direction:surface_wind_direction,mixing_height:mixing_height,lifted_index:lifted_index,haines_middle:haines_middle,haines_high:haines_high,ventilation_index:ventilation_index,cloud:cloud,precipitation:precipitation,z:z,height:height,temperature:temperature,relative_humidity:relative_humidity,wind_speed:wind_speed,wind_direction:wind_direction,time:time), it should have a geometry and a location property of name location

at org.geotools.gce.imagemosaic.catalog.GTDataStoreGranuleCatalog.checkMosaicSchema(GTDataStoreGranuleCatalog.java:296)

at org.geotools.gce.imagemosaic.catalog.GTDataStoreGranuleCatalog.createType(GTDataStoreGranuleCatalog.java:629)

at org.geotools.gce.imagemosaic.catalog.CachingDataStoreGranuleCatalog.createType(CachingDataStoreGranuleCatalog.java:103)

at org.geotools.gce.imagemosaic.RasterManager.createStore(RasterManager.java:1283)

at org.geotools.gce.imagemosaic.ImageMosaicReader.createCoverage(ImageMosaicReader.java:1129)

at org.geotools.gce.imagemosaic.ImageMosaicConfigHandler.updateConfiguration(ImageMosaicConfigHandler.java:879)

at org.geotools.gce.imagemosaic.ImageMosaicWalker.handleFile(ImageMosaicWalker.java:210)

at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker$MosaicDirectoryWalker.handleFile(ImageMosaicDirectoryWalker.java:98)

at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:367)

at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:335)

at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker$MosaicDirectoryWalker.(ImageMosaicDirectoryWalker.java:114)

at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker.run(ImageMosaicDirectoryWalker.java:196)

at org.geotools.gce.imagemosaic.Utils.createMosaic(Utils.java:360)

at org.geotools.gce.imagemosaic.Utils.checkSource(Utils.java:1250)

at org.geotools.gce.imagemosaic.ImageMosaicReader.initReaderFromURL(ImageMosaicReader.java:519)

at org.geotools.gce.imagemosaic.ImageMosaicReader.(ImageMosaicReader.java:471)

at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:483)

at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:103)

at org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1441)

at org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1369)

at org.geoserver.catalog.impl.CoverageStoreInfoImpl.getGridCoverageReader(CoverageStoreInfoImpl.java:59)

I have tried various versions of the indexer properties, all with no success in creating the data store. I am however able to make a simple NETCDF store of the mentioned file, and expose layers from it.

Any advice on how to proceed with successfully setting up an image mosaic for these NETCDF files would be greatly appreciated.

Steve Ferguson

2

Dear Steve,
How did you set the indexer?
Looking at your logs it seems to me like you have specified a wrong schema for the index.
Basically, you should setup a different schema for each NetCDF variable, depending on which dimensions are available for that variable. (It seems to me that you have setup a single schema containing all the variables in the same type).
About the 2 warnings, you can ignore it. Long story short, your variables are in Lambert Conformal projection so they are referred by X,Y dimensions/axis. However they also contain a reference to latitude and longitude 2D matrixes which aren’t handled.
(See http://docs.geoserver.org/latest/en/user/extensions/netcdf/netcdf.html#notes-on-supported-netcdfs).

Some months ago, we have setup a tool which allows to build the indexer.xml for a sample dataset and related netcdf low level index.
It is still experimental but it can be a good starting point:
http://geoserver.geo-solutions.it/downloads/releases/2.8.x-netcdf/CreateIndexer.jar

You can use it to create an indexer.xml to be tuned depending on your needs.

java -jar CreateIndexer.jar c:\data\wff[wff_153080300.nc](http://wff_153080300.nc) -p c:\data\GEOSERVER_DATA_DIR\user_projections
netcdf.projections.properties

The result, once customized, could be something like the one in attachment. (Make sure to update the indexer.xml final parameter depending on your folders/paths). I have also added a simple datastore.properties to put the mosaic index into a postgis db.
I have finally configured a mosaic on top of your dataset and it worked.

Hope this helps.
Daniele

_auxiliary.xml (4.79 KB)

datastore.properties (288 Bytes)

indexer.xml (5.82 KB)

···

On Thu, Nov 19, 2015 at 2:36 AM, Ferguson, Steve -FS <sferguson@anonymised.com> wrote:

Greetings,

I have been attempting to create an image mosaic for NETCDF files like this one (http://54.183.9.182/wff_153080300.nc) that I want to index by the time dimension in order to add additional forecast files as they become available. I am using the 2.9 SNAPSHOT version from 11/17 for all components.

I have tried to simply specify the directory within which this file sits as an image mosaic store, but this results in the following error :

19 Nov 01:01:54 WARN [netcdf.NetCDFGeoreferenceManager] - Unsupported axis: double lat(y=299, x=399);

:units = “degrees_north”;

:long_name = “latitude”;

:standard_name = “latitude”;

:_CoordinateAxisType = “Lat”;

in input: /gdata2/netcf/wff/wff_153080300.nc has been found

19 Nov 01:01:54 WARN [netcdf.NetCDFGeoreferenceManager] - Unsupported axis: double lon(y=299, x=399);

:units = “degrees_east”;

:long_name = “longitude”;

:standard_name = “longitude”;

:_CoordinateAxisType = “Lon”;

in input: /gdata2/netcf/wff/wff_153080300.nc has been found

19 Nov 01:03:28 ERROR [imagemosaic.Utils] -

java.lang.NullPointerException

19 Nov 01:03:28 WARN [gce.imagemosaic] - Failure occurred while collecting the granules

org.apache.commons.io.DirectoryWalker$CancelException: Operation Cancelled

at org.apache.commons.io.DirectoryWalker.checkIfCancelled(DirectoryWalker.java:395)

at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:368)

at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:335)

at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker$MosaicDirectoryWalker.(ImageMosaicDirectoryWalker.java:114)

at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker.run(ImageMosaicDirectoryWalker.java:196)

at org.geotools.gce.imagemosaic.Utils.createMosaic(Utils.java:360)

at org.geotools.gce.imagemosaic.Utils.checkSource(Utils.java:1250)

at org.geotools.gce.imagemosaic.ImageMosaicReader.initReaderFromURL(ImageMosaicReader.java:519)

at org.geotools.gce.imagemosaic.ImageMosaicReader.(ImageMosaicReader.java:471)

at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:483)

at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:103)

at org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1441)

at org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1369)

at org.geoserver.catalog.impl.CoverageStoreInfoImpl.getGridCoverageReader(CoverageStoreInfoImpl.java:59)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.geoserver.catalog.impl.ModificationProxy.invoke(ModificationProxy.java:146)

at com.sun.proxy.$Proxy53.getGridCoverageReader(Unknown Source)

at org.geoserver.web.data.layer.NewLayerPageProvider.getItemsInternal(NewLayerPageProvider.java:90)

at org.geoserver.web.data.layer.NewLayerPageProvider.getItems(NewLayerPageProvider.java:59)

at org.geoserver.web.wicket.GeoServerDataProvider.fullSize(GeoServerDataProvider.java:245)

I have tried to specify properties to the indexer in indexer.properties as outlined here (http://geoserver.geo-solutions.it/multidim/en/imagemosaic/index.html) but that produces the following error:

19 Nov 01:12:16 WARN [netcdf.NetCDFGeoreferenceManager] - Unsupported axis: double lat(y=299, x=399);

:units = “degrees_north”;

:long_name = “latitude”;

:standard_name = “latitude”;

:_CoordinateAxisType = “Lat”;

in input: /gdata2/netcf/wff/wff_153080300.nc has been found

19 Nov 01:12:16 WARN [netcdf.NetCDFGeoreferenceManager] - Unsupported axis: double lon(y=299, x=399);

:units = “degrees_east”;

:long_name = “longitude”;

:standard_name = “longitude”;

:_CoordinateAxisType = “Lon”;

in input: /gdata2/netcf/wff/wff_153080300.nc has been found

19 Nov 01:12:18 WARN [imagemosaic.catalog] - This granule catalog was not properly dispose as it still points to:ServiceInfo

description=Features from ShapefileDataStore

19 Nov 01:13:49 ERROR [imagemosaic.Utils] - Invalid mosaic schema SimpleFeatureTypeImpl cloud identified extends Feature(x:x,y:y,lat:lat,lon:lon,surface_temperature:surface_temperature,surface_dewpoint:surface_dewpoint,surface_rh:surface_rh,surface_wind_speed:surface_wind_speed,surface_wind_direction:surface_wind_direction,mixing_height:mixing_height,lifted_index:lifted_index,haines_middle:haines_middle,haines_high:haines_high,ventilation_index:ventilation_index,cloud:cloud,precipitation:precipitation,z:z,height:height,temperature:temperature,relative_humidity:relative_humidity,wind_speed:wind_speed,wind_direction:wind_direction,time:time), it should have a geometry and a location property of name location

java.lang.IllegalArgumentException: Invalid mosaic schema SimpleFeatureTypeImpl cloud identified extends Feature(x:x,y:y,lat:lat,lon:lon,surface_temperature:surface_temperature,surface_dewpoint:surface_dewpoint,surface_rh:surface_rh,surface_wind_speed:surface_wind_speed,surface_wind_direction:surface_wind_direction,mixing_height:mixing_height,lifted_index:lifted_index,haines_middle:haines_middle,haines_high:haines_high,ventilation_index:ventilation_index,cloud:cloud,precipitation:precipitation,z:z,height:height,temperature:temperature,relative_humidity:relative_humidity,wind_speed:wind_speed,wind_direction:wind_direction,time:time), it should have a geometry and a location property of name location

at org.geotools.gce.imagemosaic.catalog.GTDataStoreGranuleCatalog.checkMosaicSchema(GTDataStoreGranuleCatalog.java:296)

at org.geotools.gce.imagemosaic.catalog.GTDataStoreGranuleCatalog.createType(GTDataStoreGranuleCatalog.java:629)

at org.geotools.gce.imagemosaic.catalog.CachingDataStoreGranuleCatalog.createType(CachingDataStoreGranuleCatalog.java:103)

at org.geotools.gce.imagemosaic.RasterManager.createStore(RasterManager.java:1283)

at org.geotools.gce.imagemosaic.ImageMosaicReader.createCoverage(ImageMosaicReader.java:1129)

at org.geotools.gce.imagemosaic.ImageMosaicConfigHandler.updateConfiguration(ImageMosaicConfigHandler.java:879)

at org.geotools.gce.imagemosaic.ImageMosaicWalker.handleFile(ImageMosaicWalker.java:210)

at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker$MosaicDirectoryWalker.handleFile(ImageMosaicDirectoryWalker.java:98)

at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:367)

at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:335)

at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker$MosaicDirectoryWalker.(ImageMosaicDirectoryWalker.java:114)

at org.geotools.gce.imagemosaic.ImageMosaicDirectoryWalker.run(ImageMosaicDirectoryWalker.java:196)

at org.geotools.gce.imagemosaic.Utils.createMosaic(Utils.java:360)

at org.geotools.gce.imagemosaic.Utils.checkSource(Utils.java:1250)

at org.geotools.gce.imagemosaic.ImageMosaicReader.initReaderFromURL(ImageMosaicReader.java:519)

at org.geotools.gce.imagemosaic.ImageMosaicReader.(ImageMosaicReader.java:471)

at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:483)

at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:103)

at org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1441)

at org.geoserver.catalog.ResourcePool.getGridCoverageReader(ResourcePool.java:1369)

at org.geoserver.catalog.impl.CoverageStoreInfoImpl.getGridCoverageReader(CoverageStoreInfoImpl.java:59)

I have tried various versions of the indexer properties, all with no success in creating the data store. I am however able to make a simple NETCDF store of the mentioned file, and expose layers from it.

Any advice on how to proceed with successfully setting up an image mosaic for these NETCDF files would be greatly appreciated.

Steve Ferguson

Geoserver-users mailing list
Geoserver-users@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

–

==
GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.