[Geoserver-users] Any hints why these .grb2 datasets fail to load as GRIB raster stores?

Padhrig_Mccarthy · October 16, 2017, 9:03pm

Hi-

I am trying to use GeoServer 2.9.1 to create ImageMosaics from a bunch of .grb2 grids, and am having trouble with these two. I can’t seem to even read a single file of either type to create a GRIB raster store. I have looked carefully at the logs, and there appear to be no hints beyond “Failed to create reader from file.”

http://www.rap.ucar.edu/staff/paddy/ceil_20171016_v_180000_l_0000000.grb2

http://www.rap.ucar.edu/staff/paddy/fltcat_20171016_v_200000_l_0000000.grb2

I’m downloading the source now, to see if I can get some more detail about what is being logged. But this group has been a great resource in the past so I thought to give you all a try again.

Thank you in advance for any help,

-Paddy McCarthy
Research Applications Lab
National Center for Atmospheric Research
Boulder, Colorado

Bart_van_den_Eijnde1 · October 17, 2017, 12:25am

Paddy,

they both work fine for me as single-file GRIB stores (WMS output attached).

Is the directory containing the GRIB files writeable? GeoServer (and the NetCDF-Java library it uses to read GRIB files) will need to write a bunch of metadata (hidden index files) when reading NetCDF/GRIB:
http://docs.geoserver.org/latest/en/user/extensions/netcdf/netcdf.html#netcdf-files-in-read-only-directories

I do not know if this setting is honoured by NetCDF-Java for its GRIB-specific index files.

Kind regards,
Ben.

On 17/10/17 10:03, Padhrig Mccarthy wrote:

Hi-

I am trying to use GeoServer 2.9.1 to create ImageMosaics from a bunch of
.grb2 grids, and am having trouble with these two. I can't seem to even
read a single file of either type to create a GRIB raster store. I have
looked carefully at the logs, and there appear to be no hints beyond "Failed
to create reader from file."

http://www.rap.ucar.edu/staff/paddy/ceil_20171016_v_180000_l_0000000.grb2
http://www.rap.ucar.edu/staff/paddy/fltcat_20171016_v_200000_l_0000000.grb2

I'm downloading the source now, to see if I can get some more detail about
what is being logged. But this group has been a great resource in the past
so I thought to give you all a try again.

Thank you in advance for any help,

-Paddy McCarthy
  Research Applications Lab
  National Center for Atmospheric Research
  Boulder, Colorado

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/>
New Zealand

Seib_Jurgen · October 17, 2017, 7:04am

Hi Paddy,

the Grib-Plugin of the latest GeoServer version 2.12.x checks the file names in org.geotools.coverage.io.grib.GRIBFormat.

// Check if it is a GRIB data and if the GRIB library is available

boolean gribExtension = UnidataUtilities.isGribAvailable() && (fileName.contains(“grb”) || fileName.contains(“grib”));

So, the problem could simply be that your file suffix is .grb2. Rename your files either to .grb or .grib and try again.

A further issue could be that the Grib-Plugin of GeoServer 2.9.1 does not support Grib2. Try to create a GRIB raster store for a single Grib file first instead of creating an ImageMosaic.

Kind regards,

Jürgen

···

Hi-

I am trying to use GeoServer 2.9.1 to create ImageMosaics from a bunch of .grb2 grids, and am having trouble with these two. I can’t seem to even read a single file of either type to create a GRIB raster store. I have looked carefully at the logs, and there appear to be no hints beyond “Failed to create reader from file.”

http://www.rap.ucar.edu/staff/paddy/ceil_20171016_v_180000_l_0000000.grb2

http://www.rap.ucar.edu/staff/paddy/fltcat_20171016_v_200000_l_0000000.grb2

I’m downloading the source now, to see if I can get some more detail about what is being logged. But this group has been a great resource in the past so I thought to give you all a try again.

Thank you in advance for any help,

-Paddy McCarthy

Research Applications Lab

National Center for Atmospheric Research

Boulder, Colorado

Bart_van_den_Eijnde1 · October 17, 2017, 12:03pm

On 17/10/17 20:04, Seib Jürgen wrote:

the Grib-Plugin of the latest GeoServer version 2.12.x checks the file names in org.geotools.coverage.io.grib.GRIBFormat.
// Check if it is a GRIB data and if the GRIB library is available
boolean gribExtension = UnidataUtilities.isGribAvailable() && (fileName.contains("grb") || fileName.contains("grib"));
So, the problem could simply be that your file suffix is .grb2. Rename your files either to .grb or .grib and try again.

I think this is unlikely to be the problem because fileName.contains("grb") will be true for filenames that end with ".grb2".

This is still a useful point: for filenames that do not contain "grb" or "grib", under Linux a symbolic link containing one of these can be used.

A further issue could be that the Grib-Plugin of GeoServer 2.9.1 does not support Grib2.

I can assure you that the GRIB plugin of GeoServer 2.9.1 most definitely supports GRIB2. I used GeoServer 2.9.1 (built on GeoTools 15.1) for GRIB2 single file and ImageMosaic:
https://osgeo-org.atlassian.net/browse/GEOT-5434

Kind regards,

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/>
New Zealand

Padhrig_Mccarthy · October 18, 2017, 4:53pm

Thank you all for your suggestions. They encouraged me to dig deeper into our problem.

The issue ended up being that the National Weather Service (the source of these two grids) is adding a 32-byte header to the files. GeoTools checks that grib files start with the characters GRIB, which was no longer the case with the prepended header.

We had to connect to our GeoServer instance via a remote debugger to figure out the problem. If anyone is curious about how to do that, please contact me. It was challenging to figure out, but I’m sure will prove helpful as we work with GeoServer more.

Thanks again for your help.

-Paddy McCarthy
Research Applications Lab
National Center for Atmospheric Research
Boulder, Colorado

···

On Tue, Oct 17, 2017 at 8:03 AM, Ben Caradoc-Davies <ben@anonymised.com> wrote:

On 17/10/17 20:04, Seib Jürgen wrote:

the Grib-Plugin of the latest GeoServer version 2.12.x checks the file names in org.geotools.coverage.io.grib.GRIBFormat.
// Check if it is a GRIB data and if the GRIB library is available
boolean gribExtension = UnidataUtilities.isGribAvailable() && (fileName.contains(“grb”) || fileName.contains(“grib”));
So, the problem could simply be that your file suffix is .grb2. Rename your files either to .grb or .grib and try again.

I think this is unlikely to be the problem because fileName.contains(“grb”) will be true for filenames that end with “.grb2”.

This is still a useful point: for filenames that do not contain “grb” or “grib”, under Linux a symbolic link containing one of these can be used.

A further issue could be that the Grib-Plugin of GeoServer 2.9.1 does not support Grib2.

I can assure you that the GRIB plugin of GeoServer 2.9.1 most definitely supports GRIB2. I used GeoServer 2.9.1 (built on GeoTools 15.1) for GRIB2 single file and ImageMosaic:
https://osgeo-org.atlassian.net/browse/GEOT-5434

Kind regards,

–
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/>
New Zealand

Bart_van_den_Eijnde1 · October 18, 2017, 7:46pm

Paddy,

is this the World Meteorological Organization (WMO) header added by the US National Weather Service Telecommunications Gateway? If so, this is fixed in GeoTools 15.2 / GeoServer 2.9.2 and later:

[GEOT-5524] Incorrect detection of GRIB files with WMO header
https://osgeo-org.atlassian.net/browse/GEOT-5524

Kind regards,
Ben.

On 19/10/17 05:53, Padhrig Mccarthy wrote:

Thank you all for your suggestions. They encouraged me to dig deeper into
our problem.

The issue ended up being that the National Weather Service (the source of
these two grids) is adding a 32-byte header to the files. GeoTools checks
that grib files start with the characters GRIB, which was no longer the
case with the prepended header.

We had to connect to our GeoServer instance via a remote debugger to figure
out the problem. If anyone is curious about how to do that, please contact
me. It was challenging to figure out, but I'm sure will prove helpful as we
work with GeoServer more.

Thanks again for your help.

-Paddy McCarthy
  Research Applications Lab
  National Center for Atmospheric Research
  Boulder, Colorado

--
Ben Caradoc-Davies <ben@anonymised.com>
Director
Transient Software Limited <http://transient.nz/>
New Zealand