[Geoserver-users] ImageMosaic custom dimension internal to NetCDF files

I have several NetCDF files that comprise an ImageMosaic time series. This works fine for variables with only spatial dimensions, and I can get NetCDF output from WCS:

foo(y, x) --ImageMosaic-NetCDF-out--> foo(t, y, x)

However, when there is an extra dimension in the source NetCDF files (same values in all files), it is silently dropped:

bar(s, y, x) --ImageMosaic-NetCDF-out--> bar(t, y, x)

Not, as I had hoped:

bar(s, y, x) --ImageMosaic-NetCDF-out--> bar(t, s, y, x)

The documentation describes mosaicking over a custom dimension which varies between files and whose value is stored in the granule index. How to I deliver a variable with a dimension that is internal to each source NetCDF file? Is this supported? Is extra configuration is required?

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

I Ben.
I’m not sure to have fully understood your problem/question.
Can you give us an example of a ncdump output (a shorter form of the interesting part) for 2 involved files?

As far as I remember, the current NetCDF low level readers only deal with inner dimension referencing to times and elevations/depths/heights/ (everything is a Z).

Please, let us know.
Cheers,
Daniele

···

On Sat, May 28, 2016 at 6:45 AM, Ben Caradoc-Davies <ben@anonymised.com> wrote:

I have several NetCDF files that comprise an ImageMosaic time series.
This works fine for variables with only spatial dimensions, and I can
get NetCDF output from WCS:

foo(y, x) --ImageMosaic-NetCDF-out–> foo(t, y, x)

However, when there is an extra dimension in the source NetCDF files
(same values in all files), it is silently dropped:

bar(s, y, x) --ImageMosaic-NetCDF-out–> bar(t, y, x)

Not, as I had hoped:

bar(s, y, x) --ImageMosaic-NetCDF-out–> bar(t, s, y, x)

The documentation describes mosaicking over a custom dimension which
varies between files and whose value is stored in the granule index. How
to I deliver a variable with a dimension that is internal to each source
NetCDF file? Is this supported? Is extra configuration is required?

Kind regards,


Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/>
New Zealand


What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e


Geoserver-users mailing list
Geoserver-users@anonymised.comsts.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it


AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

Hi all

I have a similar (I think) issue I’d like to solve. Like Ben, I have netCDF files with bands, and in each band are timeslices. Here’s an openDAP look inside a sample file:

http://dapds01.nci.org.au/thredds/dodsC/uc0/rs0_dev/alberscube/LANDSAT_5_TM_NBAR_ALB_15_-41_1990-03-02T23-11-39.000000.nc.das

If I publish each band as a layer (eg band1, band30, …), I can see the time slices inside each ‘band’. What I really *want* to do is publish the netCDF file as one unit - so that an end user can subset the data with all the bands and time slices intact, or potentially do some band maths at the server side (using WPS).

On a similar topic (without wanting to derail this conversation) - does geoserver have any /plan to have any openDAP style services? It’s great to be able to subset netCDFs, but being able to query a file (like the link above) before downloading it would be great!

Thanks

Adam

On 30 May 2016, at 5:40 PM, Daniele Romagnoli <daniele.romagnoli@...1107...> wrote:

I Ben.
I'm not sure to have fully understood your problem/question.
Can you give us an example of a ncdump output (a shorter form of the interesting part) for 2 involved files?

As far as I remember, the current NetCDF low level readers only deal with inner dimension referencing to times and elevations/depths/heights/ (everything is a Z).

Please, let us know.
Cheers,
Daniele

On Sat, May 28, 2016 at 6:45 AM, Ben Caradoc-Davies <ben@...6881...> wrote:
I have several NetCDF files that comprise an ImageMosaic time series.
This works fine for variables with only spatial dimensions, and I can
get NetCDF output from WCS:

foo(y, x) --ImageMosaic-NetCDF-out--> foo(t, y, x)

However, when there is an extra dimension in the source NetCDF files
(same values in all files), it is silently dropped:

bar(s, y, x) --ImageMosaic-NetCDF-out--> bar(t, y, x)

Not, as I had hoped:

bar(s, y, x) --ImageMosaic-NetCDF-out--> bar(t, s, y, x)

The documentation describes mosaicking over a custom dimension which
varies between files and whose value is stored in the granule index. How
to I deliver a variable with a dimension that is internal to each source
NetCDF file? Is this supported? Is extra configuration is required?

Kind regards,

--
Ben Caradoc-Davies <ben@...6881...>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------
AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

Daniele,

these dimensions are in effect a Z dimension. The data is sourced from GRIB2 but by the time it is ingested by ImageMosaic, it is equivalent to NetCDF (CDM API). Below I have included a small but self-consistent subset from the CDL generated with NetCDF-Java ToolsUI (current master 4.6.6-SNAPSHOT with rotated pole GRIB2 support). There are multiple files that vary in time only, and ImageMosaic aggregates them into a time series. I have included two variables as they appear in one file (granule):

Temperature_surface(time=1, rlat=834, rlon=953) works fine and I can use ImageMosaic to obtain a correct WCS time series. Panoply animations here:
https://github.com/bencaradocdavies/geoserver/wiki/RAP-Native-Grid-ImageMosaic

u-component_of_wind_sigma(time=1, sigma=50, rlat=834, rlon=953) appears to work but the output contains no sigma dimension. If I add a sigma dimension in indexer.xml, the ImageMosaic refuses to load. Without a sigma dimension in indexer.xml, the request appears to work, takes 50 times longer (as expected), but the output file contains no sigma dimension or coordinate variable and is the wrong size (same as Temperature_surface).

In a nutshell: should elevation variables that are *internal* to each granule be indexed? Is this supported? If so, is indexer configuration required? Why would the granule index need to know?

The indexer domain I tried unsuccessfully was this (no entry in the granule schema, ImageMosaic would not load):

<domain name="sigma">
<attributes>
<attribute>sigma</attribute>
</attributes>
</domain>

All the examples that I have seen with an elevation or custom dimension have been where this dimension has a single value for each granule and varies across them. I have not seen an example with extra dimensions internal to each NetCDF file (granule).

Kind regards,
Ben.

netcdf forecast {
   dimensions:
     rlat = 834;
     rlon = 953;
     time = 1;
     sigma = 50;
   variables:
     int RotatedLatLon32769_Projection;
       :grid_mapping_name = "rotated_latitude_longitude";
       :grid_north_pole_latitude = 36.0; // double
       :grid_north_pole_longitude = 74.0; // double
     float rlat(rlat=834);
       :standard_name = "grid_latitude";
       :units = "degrees";
     float rlon(rlon=953);
       :standard_name = "grid_longitude";
       :units = "degrees";
     double reftime;
       :units = "Hour since 2016-04-25T22:00:00Z";
       :standard_name = "forecast_reference_time";
       :long_name = "GRIB reference time";
       :calendar = "proleptic_gregorian";
     double time(time=1);
       :units = "Hour since 2016-04-25T22:00:00Z";
       :standard_name = "time";
       :long_name = "GRIB forecast or observation time";
       :calendar = "proleptic_gregorian";
     float sigma(sigma=50);
       :units = "sigma";
       :long_name = "Sigma level";
       :positive = "down";
       :Grib_level_type = 104; // int
     float Temperature_surface(time=1, rlat=834, rlon=953);
       :long_name = "Temperature @ Ground or water surface";
       :units = "K";
       :abbreviation = "TMP";
       :missing_value = NaNf; // float
       :grid_mapping = "RotatedLatLon32769_Projection";
       :coordinates = "reftime time rlat rlon ";
       :Grib_Variable_Id = "VAR_0-0-0_L1";
       :Grib2_Parameter = 0, 0, 0; // int
       :Grib2_Parameter_Discipline = "Meteorological products";
       :Grib2_Parameter_Category = "Temperature";
       :Grib2_Parameter_Name = "Temperature";
       :Grib2_Level_Type = "Ground or water surface";
       :Grib2_Generating_Process_Type = "Forecast";
     float u-component_of_wind_sigma(time=1, sigma=50, rlat=834, rlon=953);
       :long_name = "u-component of wind @ Sigma level";
       :units = "m/s";
       :abbreviation = "UGRD";
       :missing_value = NaNf; // float
       :grid_mapping = "RotatedLatLon32769_Projection";
       :coordinates = "reftime time sigma rlat rlon ";
       :Grib_Variable_Id = "VAR_0-2-2_L104";
       :Grib2_Parameter = 0, 2, 2; // int
       :Grib2_Parameter_Discipline = "Meteorological products";
       :Grib2_Parameter_Category = "Momentum";
       :Grib2_Parameter_Name = "u-component of wind";
       :Grib2_Level_Type = "Sigma level";
       :Grib2_Generating_Process_Type = "Forecast";
   // global attributes:
   :Originating_or_generating_Center = "The NOAA Forecast Systems Laboratory, Boulder, CO, United States";
   :Originating_or_generating_Subcenter = "0";
   :GRIB_table_version = "2,1";
   :Type_of_generating_process = "Forecast";
:Analysis_or_forecast_generating_process_identifier_defined_by_originating_centre = "High-Resolution Rapid Refresh";
   :file_format = "GRIB-2";
   :Conventions = "CF-1.6";
   :history = "Read using CDM IOSP GribCollection v3";
   :featureType = "GRID";
}

On 30/05/16 19:40, Daniele Romagnoli wrote:

I Ben.
I'm not sure to have fully understood your problem/question.
Can you give us an example of a ncdump output (a shorter form of the
interesting part) for 2 involved files?

As far as I remember, the current NetCDF low level readers only deal with
inner dimension referencing to times and elevations/depths/heights/
(everything is a Z).

Please, let us know.
Cheers,
Daniele

On Sat, May 28, 2016 at 6:45 AM, Ben Caradoc-Davies <ben@anonymised.com>
wrote:

I have several NetCDF files that comprise an ImageMosaic time series.
This works fine for variables with only spatial dimensions, and I can
get NetCDF output from WCS:

foo(y, x) --ImageMosaic-NetCDF-out--> foo(t, y, x)

However, when there is an extra dimension in the source NetCDF files
(same values in all files), it is silently dropped:

bar(s, y, x) --ImageMosaic-NetCDF-out--> bar(t, y, x)

Not, as I had hoped:

bar(s, y, x) --ImageMosaic-NetCDF-out--> bar(t, s, y, x)

The documentation describes mosaicking over a custom dimension which
varies between files and whose value is stored in the granule index. How
to I deliver a variable with a dimension that is internal to each source
NetCDF file? Is this supported? Is extra configuration is required?

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and
traffic
patterns at an interface-level. Reveals which users, apps, and protocols
are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

Hi Ben,

···

On Tue, May 31, 2016 at 4:25 AM, Ben Caradoc-Davies <ben@anonymised.com.> wrote:

Daniele,

these dimensions are in effect a Z dimension. The data is sourced from GRIB2 but by the time it is ingested by ImageMosaic, it is equivalent to NetCDF (CDM API). Below I have included a small but self-consistent subset from the CDL generated with NetCDF-Java ToolsUI (current master 4.6.6-SNAPSHOT with rotated pole GRIB2 support). There are multiple files that vary in time only, and ImageMosaic aggregates them into a time series. I have included two variables as they appear in one file (granule):

Temperature_surface(time=1, rlat=834, rlon=953) works fine and I can use ImageMosaic to obtain a correct WCS time series. Panoply animations here:
https://github.com/bencaradocdavies/geoserver/wiki/RAP-Native-Grid-ImageMosaic

u-component_of_wind_sigma(time=1, sigma=50, rlat=834, rlon=953) appears to work but the output contains no sigma dimension. If I add a sigma dimension in indexer.xml, the ImageMosaic refuses to load. Without a sigma dimension in indexer.xml, the request appears to work, takes 50 times longer (as expected), but the output file contains no sigma dimension or coordinate variable and is the wrong size (same as Temperature_surface).

In a nutshell: should elevation variables that are internal to each granule be indexed? Is this supported? If so, is indexer configuration required? Why would the granule index need to know?

I would say yes. In the past we have worked with NetCDF containing sigma levels, exposing them as sigma dimension, similarly to what you did.
If my memory serves me right, when having issues with custom dimensions the problem was related to the axisType not being declared but I should double check.
Would this help?

Daniele

The indexer domain I tried unsuccessfully was this (no entry in the granule schema, ImageMosaic would not load):

sigma

All the examples that I have seen with an elevation or custom dimension have been where this dimension has a single value for each granule and varies across them. I have not seen an example with extra dimensions internal to each NetCDF file (granule).

Kind regards,
Ben.

netcdf forecast {
dimensions:
rlat = 834;
rlon = 953;
time = 1;
sigma = 50;
variables:
int RotatedLatLon32769_Projection;
:grid_mapping_name = “rotated_latitude_longitude”;
:grid_north_pole_latitude = 36.0; // double
:grid_north_pole_longitude = 74.0; // double
float rlat(rlat=834);
:standard_name = “grid_latitude”;
:units = “degrees”;
float rlon(rlon=953);
:standard_name = “grid_longitude”;
:units = “degrees”;
double reftime;
:units = “Hour since 2016-04-25T22:00:00Z”;
:standard_name = “forecast_reference_time”;
:long_name = “GRIB reference time”;
:calendar = “proleptic_gregorian”;
double time(time=1);
:units = “Hour since 2016-04-25T22:00:00Z”;
:standard_name = “time”;
:long_name = “GRIB forecast or observation time”;
:calendar = “proleptic_gregorian”;
float sigma(sigma=50);
:units = “sigma”;
:long_name = “Sigma level”;
:positive = “down”;
:Grib_level_type = 104; // int
float Temperature_surface(time=1, rlat=834, rlon=953);
:long_name = “Temperature @ Ground or water surface”;
:units = “K”;
:abbreviation = “TMP”;
:missing_value = NaNf; // float
:grid_mapping = “RotatedLatLon32769_Projection”;
:coordinates = "reftime time rlat rlon ";
:Grib_Variable_Id = “VAR_0-0-0_L1”;
:Grib2_Parameter = 0, 0, 0; // int
:Grib2_Parameter_Discipline = “Meteorological products”;
:Grib2_Parameter_Category = “Temperature”;
:Grib2_Parameter_Name = “Temperature”;
:Grib2_Level_Type = “Ground or water surface”;
:Grib2_Generating_Process_Type = “Forecast”;
float u-component_of_wind_sigma(time=1, sigma=50, rlat=834, rlon=953);
:long_name = “u-component of wind @ Sigma level”;
:units = “m/s”;
:abbreviation = “UGRD”;
:missing_value = NaNf; // float
:grid_mapping = “RotatedLatLon32769_Projection”;
:coordinates = "reftime time sigma rlat rlon ";
:Grib_Variable_Id = “VAR_0-2-2_L104”;
:Grib2_Parameter = 0, 2, 2; // int
:Grib2_Parameter_Discipline = “Meteorological products”;
:Grib2_Parameter_Category = “Momentum”;
:Grib2_Parameter_Name = “u-component of wind”;
:Grib2_Level_Type = “Sigma level”;
:Grib2_Generating_Process_Type = “Forecast”;
// global attributes:
:Originating_or_generating_Center = “The NOAA Forecast Systems Laboratory, Boulder, CO, United States”;
:Originating_or_generating_Subcenter = “0”;
:GRIB_table_version = “2,1”;
:Type_of_generating_process = “Forecast”;

:Analysis_or_forecast_generating_process_identifier_defined_by_originating_centre = “High-Resolution Rapid Refresh”;
:file_format = “GRIB-2”;
:Conventions = “CF-1.6”;
:history = “Read using CDM IOSP GribCollection v3”;
:featureType = “GRID”;

}

On 30/05/16 19:40, Daniele Romagnoli wrote:

I Ben.
I’m not sure to have fully understood your problem/question.
Can you give us an example of a ncdump output (a shorter form of the
interesting part) for 2 involved files?

As far as I remember, the current NetCDF low level readers only deal with
inner dimension referencing to times and elevations/depths/heights/
(everything is a Z).

Please, let us know.
Cheers,
Daniele

On Sat, May 28, 2016 at 6:45 AM, Ben Caradoc-Davies <ben@anonymised.com>
wrote:

I have several NetCDF files that comprise an ImageMosaic time series.
This works fine for variables with only spatial dimensions, and I can
get NetCDF output from WCS:

foo(y, x) --ImageMosaic-NetCDF-out–> foo(t, y, x)

However, when there is an extra dimension in the source NetCDF files
(same values in all files), it is silently dropped:

bar(s, y, x) --ImageMosaic-NetCDF-out–> bar(t, y, x)

Not, as I had hoped:

bar(s, y, x) --ImageMosaic-NetCDF-out–> bar(t, s, y, x)

The documentation describes mosaicking over a custom dimension which
varies between files and whose value is stored in the granule index. How
to I deliver a variable with a dimension that is internal to each source
NetCDF file? Is this supported? Is extra configuration is required?

Kind regards,


Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/>
New Zealand


What NetFlow Analyzer can do for you? Monitors network bandwidth and
traffic
patterns at an interface-level. Reveals which users, apps, and protocols
are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e


Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users


Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/>
New Zealand

==

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it


AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

On 01/06/16 00:26, Daniele Romagnoli wrote:

On Tue, May 31, 2016 at 4:25 AM, Ben Caradoc-Davies <ben@anonymised.com>
wrote:

In a nutshell: should elevation variables that are *internal* to each
granule be indexed? Is this supported? If so, is indexer configuration
required? Why would the granule index need to know?

I would say yes. In the past we have worked with NetCDF containing sigma
levels, exposing them as sigma dimension, similarly to what you did.
If my memory serves me right, when having issues with custom dimensions the
problem was related to the axisType not being declared but I should double
check.
Would this help?

Yes, please.

- When using a custom dimension like sigma, should it be specified in the indexer.xml?

- If an indexer dimension for sigma is required, what form should it take?

- Does sigma need an entry in the granule schema in indexer.xml?

- How did you overcome sigma not having a recognised axisType?

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

On Tue, May 31, 2016 at 11:06 PM, Ben Caradoc-Davies <ben@anonymised.com>
wrote:

On 01/06/16 00:26, Daniele Romagnoli wrote:

On Tue, May 31, 2016 at 4:25 AM, Ben Caradoc-Davies <ben@anonymised.com>
wrote:

In a nutshell: should elevation variables that are *internal* to each
granule be indexed? Is this supported? If so, is indexer configuration
required? Why would the granule index need to know?

I would say yes. In the past we have worked with NetCDF containing sigma
levels, exposing them as sigma dimension, similarly to what you did.
If my memory serves me right, when having issues with custom dimensions
the
problem was related to the axisType not being declared but I should double
check.
Would this help?

Yes, please.

- When using a custom dimension like sigma, should it be specified in the
indexer.xml?

Yes. each domain/dimension involved in your mosaic should be declared.

- If an indexer dimension for sigma is required, what form should it take?

You should declare it in the <domains> section.
As an instance
<domains>
   <domain name="sigma"> <!-- See Note1 -->
      <attributes><attribute>sigma</attribute></attributes> <!-- See Note2
-->
   </domain>

Note1: you can put any name you want here. It will represent the dimension
exposed in GeoServer.
Note2: the attribute here should match the real dimension name on the
underlying NetCDF dataset.

- Does sigma need an entry in the granule schema in indexer.xml?

Yes.
Your <coverage> should refer that sigma domain as well as include a column
in the schema
<coverage>
   <name>u-component_of_wind_sigma</name>
   <domains>
      .... time?
      <domain ref="sigma" />
   </domains>
    <schema name="u-component_of_wind_sigma" >
        <attributes>.........., sigma: java.lang.Float</attributes> <!--
See Note 3 -->
    </schema>

Note 3: the attribute in the schema should match the attribute of the
previously declared domain.

Note that imageMosaic has its own indexer. NetCDFs are multidimensional
datasets so they also have a similar indexer to catalog NetCDF sections
into a (by default) H2 hidden DB. That inner indexer is generated on the
fly OR you can define it once (in case all NetCDF has same structure) and
declare it in the imageMosaic indexer.xml
http://geoserver.geo-solutions.it/multidim/en/netcdf/netcdf_indexer.html#netcdf-indexer-file

How your NetCDF indexer file looks like?
Some months ago I have developed an experimental tool
<http://geoserver.geo-solutions.it/downloads/releases/2.8.x-netcdf/CreateIndexer.jar&gt;
which may help in setting up the indexers:

*java -jar CreateIndexer.jar /path/to/a/sample.nc <http://sample.nc> -p
/path/to/GEOSERVER_DATA_DIR/**user_projections/*
*netcdf.projections.properties*

Notes
<http://docs.geoserver.org/stable/en/user/extensions/netcdf/netcdf.html#supporting-custom-netcdf-coordinate-reference-systems&gt;
on netcdf.projections.properties.

It will create both the ImageMosaic indexer.xml and the NetCDF indexer for
a sample.nc file of a collection of NetCDF files with same structure. You
can take a look on the produced results and customize them if needed.
As I said, that tool is still experimental and it can be surely improved or
may throws errors on some untested datasets.

- How did you overcome sigma not having a recognised axisType?

If my memory serves me right the data producer fixed their coordinate
variables but I'm not pretty sure about this since it was the past year.
Let's see if the previous suggestions allows you to get them work.
Otherwise, you may consider sharing a sample or do some debug on the low
level readers to check if that's the case.

Cheers,
Daniele

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

On 01/06/16 20:12, Daniele Romagnoli wrote:

On Tue, May 31, 2016 at 11:06 PM, Ben Caradoc-Davies <ben@anonymised.com>

- When using a custom dimension like sigma, should it be specified in the
indexer.xml?

Yes. each domain/dimension involved in your mosaic should be declared.

- If an indexer dimension for sigma is required, what form should it take?

You should declare it in the <domains> section.
As an instance
<domains>
   <domain name="sigma"> <!-- See Note1 -->
      <attributes><attribute>sigma</attribute></attributes> <!-- See Note2
-->
   </domain>
Note1: you can put any name you want here. It will represent the dimension
exposed in GeoServer.
Note2: the attribute here should match the real dimension name on the
underlying NetCDF dataset.

- Does sigma need an entry in the granule schema in indexer.xml?

Yes.
Your <coverage> should refer that sigma domain as well as include a column
in the schema
<coverage>
   <name>u-component_of_wind_sigma</name>
   <domains>
      .... time?
      <domain ref="sigma" />
   </domains>
    <schema name="u-component_of_wind_sigma" >
        <attributes>.........., sigma: java.lang.Float</attributes> <!--
See Note 3 -->
    </schema>
Note 3: the attribute in the schema should match the attribute of the
previously declared domain.

Thank you so much, Daniele. That works almost perfectly.

The one problem is that Float is not supported, only times, Integer, Double, and String (and unsupported types like Float are treated as String and fail because String cannot be a coordinate variable type):
https://github.com/geotools/geotools/blob/master/modules/plugin/coverage-multidim/netcdf/src/main/java/org/geotools/imageio/netcdf/utilities/NetCDFUtilities.java#L852

If I add a sigma domain as you described above and a schema attribute "sigma:Double", then enable both time and sigma in the Dimensions tab in the web UI, the coverage is returned with both time and sigma dimensions as expected. I also had to increase my heap to -Xmx2048m. I am now looking at a 647MB NetCDF-4 file with 4 time values and 50 sigma layers in Panoply. :slight_smile:

Thanks also for your other suggestions. I have not yet tried them.

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

On Thu, Jun 2, 2016 at 3:28 AM, Ben Caradoc-Davies <ben@anonymised.com> wrote:

On 01/06/16 20:12, Daniele Romagnoli wrote:

On Tue, May 31, 2016 at 11:06 PM, Ben Caradoc-Davies <ben@anonymised.com>

- When using a custom dimension like sigma, should it be specified in the
indexer.xml?

Yes. each domain/dimension involved in your mosaic should be declared.

- If an indexer dimension for sigma is required, what form should it
take?

You should declare it in the <domains> section.
As an instance
<domains>
   <domain name="sigma"> <!-- See Note1 -->
      <attributes><attribute>sigma</attribute></attributes> <!-- See Note2
-->
   </domain>
Note1: you can put any name you want here. It will represent the dimension
exposed in GeoServer.
Note2: the attribute here should match the real dimension name on the
underlying NetCDF dataset.

- Does sigma need an entry in the granule schema in indexer.xml?

Yes.
Your <coverage> should refer that sigma domain as well as include a column
in the schema
<coverage>
   <name>u-component_of_wind_sigma</name>
   <domains>
      .... time?
      <domain ref="sigma" />
   </domains>
    <schema name="u-component_of_wind_sigma" >
        <attributes>.........., sigma: java.lang.Float</attributes> <!--
See Note 3 -->
    </schema>
Note 3: the attribute in the schema should match the attribute of the
previously declared domain.

Thank you so much, Daniele. That works almost perfectly.

The one problem is that Float is not supported, only times, Integer,
Double, and String (and unsupported types like Float are treated as String
and fail because String cannot be a coordinate variable type):

https://github.com/geotools/geotools/blob/master/modules/plugin/coverage-multidim/netcdf/src/main/java/org/geotools/imageio/netcdf/utilities/NetCDFUtilities.java#L852

Mmm... I should already have fixed that several months ago. I'll double

check it again.

If I add a sigma domain as you described above and a schema attribute
"sigma:Double", then enable both time and sigma in the Dimensions tab in
the web UI, the coverage is returned with both time and sigma dimensions as
expected. I also had to increase my heap to -Xmx2048m. I am now looking at
a 647MB NetCDF-4 file with 4 time values and 50 sigma layers in Panoply.
:slight_smile:

Good news :slight_smile:

Cheers,
Daniele

Thanks also for your other suggestions. I have not yet tried them.

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/&gt;
New Zealand

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Daniele Romagnoli
Senior Software Engineer

GeoSolutions S.A.S.
Via di Montramito 3/A
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.