[Geoserver-devel] Modification to support custom dimensions for coverage layers

Hello,

I am working on some modifications to allow GeoServer to support dimensions
other than TIME and ELEVATION for coverage layers, and I was wondering if
this group could comment on whether this approach is reasonable, and also
what it would take to get my changes included in the mainline GeoServer
branch.

As I understand it, GeoServer currently supports TIME and ELEVATION
dimensions for coverage layers through the use of custom plugins that define
their own coverage readers (by subclassing
org.geotools.coverage.grid.io.AbstractGridCoverage2DReader). The actual
handling of dimensions is determined by the custom coverage reader's
implementation of the getMetadataValue(String) and
read(GeneralParameterValue) methods, while the set of supported dimensions
is advertised via its getMetadataNames() method. This mechanism is currently
limited strictly by the fact that the core GeoServer logic expressly ignores
any metadata entry that is not a "TIME" or "ELEVATION" entry. I would like
to remove this limitation, so that custom plugins can truly support custom
dimensions.

The changes to core GeoServer would be minimal, and would occur in the
following classes:

org.geoserver.wms.capabilities.DimensionHelper (writes dimension entries to
WMS GetCapabilities response) - modify handleRasterLayerDimensions and
declareWMS11Dimensions methods to process any metadata entry whose name is
valid according to the coverage reader (in addition to the current
processing for "time" and "elevation" entries)

org.geoserver.catalog.util.ReaderDimensionsAccessor - add getDomain(String)
and hasDomain(String) methods so that coverage reader queries are not
limited to just time and elevation

DefaultWebCoverageService100 - modify getCoverage method so that in addition
to "ELEVATION" axis subsets being handled, any axis subset whose name is
valid according to the coverage reader will be similarly handled (and
corrsponding entries will be included in the array passed to the coverage
reader's read(GeneralParameterValue) method).

Any comments, as well as guidance on submitting this for inclusion in
mainline, would be appreciated.

Thank you,
Mike

--
View this message in context: http://osgeo-org.1560.n6.nabble.com/Modification-to-support-custom-dimensions-for-coverage-layers-tp4474094p4474094.html
Sent from the GeoServer - Dev mailing list archive at Nabble.com.

On Wed, Feb 15, 2012 at 10:54 PM, Mike Benowitz <michael.benowitz@anonymised.com> wrote:

Hello,

I am working on some modifications to allow GeoServer to support dimensions
other than TIME and ELEVATION for coverage layers, and I was wondering if
this group could comment on whether this approach is reasonable, and also
what it would take to get my changes included in the mainline GeoServer
branch.

As I understand it, GeoServer currently supports TIME and ELEVATION
dimensions for coverage layers through the use of custom plugins that define
their own coverage readers (by subclassing
org.geotools.coverage.grid.io.AbstractGridCoverage2DReader). The actual
handling of dimensions is determined by the custom coverage reader’s
implementation of the getMetadataValue(String) and
read(GeneralParameterValue) methods, while the set of supported dimensions
is advertised via its getMetadataNames() method. This mechanism is currently
limited strictly by the fact that the core GeoServer logic expressly ignores
any metadata entry that is not a “TIME” or “ELEVATION” entry. I would like
to remove this limitation, so that custom plugins can truly support custom
dimensions.

The thing is, the metadata values can be anything, not just dimensions.
TIME and ELEVATION are conventionally understood, if you come up with
another dimension there must be a way to know it is a dimension and not
other random metadata.

It could be that a api change is required to declare the available dimensions,
or it could be that we choose a naming convention, any metadata key
named like DIMENSION_ is to be considered a dimension.

Another thing that might problematic, but I’m not sure if it will be problematic
or not, is that we know the data type of time and elevation, but what about
the other dimensions? We treat them as strings and that’s it?

The changes to core GeoServer would be minimal, and would occur in the
following classes:

org.geoserver.wms.capabilities.DimensionHelper (writes dimension entries to
WMS GetCapabilities response) - modify handleRasterLayerDimensions and
declareWMS11Dimensions methods to process any metadata entry whose name is
valid according to the coverage reader (in addition to the current
processing for “time” and “elevation” entries)

org.geoserver.catalog.util.ReaderDimensionsAccessor - add getDomain(String)
and hasDomain(String) methods so that coverage reader queries are not
limited to just time and elevation

Seems reasonable

DefaultWebCoverageService100 - modify getCoverage method so that in addition
to “ELEVATION” axis subsets being handled, any axis subset whose name is
valid according to the coverage reader will be similarly handled (and
corrsponding entries will be included in the array passed to the coverage
reader’s read(GeneralParameterValue) method).

Might work, but I believe (not sure) that the custom dimension would be numeric
to be handled that way?

Cheers
Andrea

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf


Andrea,

Thanks for your reply. Regarding the three concerns you expressed:

1) I think distinguishing dimension metadata from other random metadata can
be handled without changing the API or relying on a naming convention. From
what I can tell, dimension entries in the coverage.xml file are
distinguished by the dimensionInfo element, which gets translated to a
DimensionInfo (or more specifically,
org.geoserver.catalog.impl.DimensionInfoImpl) value in the corresponding
entry in the MetadataMap received by DimensionHelper. The modified
DimensionHelper should be able to scan the MetadataMap for all entries whose
values are assignable to type DimensionInfo, and process only those entries
as dimensions.

2) It's true that the core logic would not be able to assume the data type
for non-standard dimensions, but this does not seem particularly
problematic. We could include processing to recognize whether strings are
numeric and if so convert them to Double, or whether they are in a
recognizable date/time format and if so convert them to Date. In any case,
it seems reasonable that values that are not in a recognized format would
just be treated as strings, and the interpretation left to the custom
plugin/client.

3) I believe the modified DefaultWebCoverageService100.getCoverage method
can handle custom dimensions regardless of whether they are numeric. As it
is now, the axis subset list items (created from the axisSubset elements in
the request document) that are processed by this method are of type
TypedLiteralType, which specifies all values as strings. The ELEVATION
processing assumes that the values can be parsed to numbers, but I don't see
anything requiring the custom dimension processing to make such an
assumption.

--
View this message in context: http://osgeo-org.1560.n6.nabble.com/Modification-to-support-custom-dimensions-for-coverage-layers-tp4474094p4478043.html
Sent from the GeoServer - Dev mailing list archive at Nabble.com.

On Fri, Feb 17, 2012 at 12:19 AM, Mike Benowitz <michael.benowitz@anonymised.com> wrote:

Andrea,

Thanks for your reply. Regarding the three concerns you expressed:

  1. I think distinguishing dimension metadata from other random metadata can
    be handled without changing the API or relying on a naming convention. From
    what I can tell, dimension entries in the coverage.xml file are
    distinguished by the dimensionInfo element, which gets translated to a
    DimensionInfo (or more specifically,
    org.geoserver.catalog.impl.DimensionInfoImpl) value in the corresponding
    entry in the MetadataMap received by DimensionHelper. The modified
    DimensionHelper should be able to scan the MetadataMap for all entries whose
    values are assignable to type DimensionInfo, and process only those entries
    as dimensions.
  1. It’s true that the core logic would not be able to assume the data type
    for non-standard dimensions, but this does not seem particularly
    problematic. We could include processing to recognize whether strings are
    numeric and if so convert them to Double, or whether they are in a
    recognizable date/time format and if so convert them to Date. In any case,
    it seems reasonable that values that are not in a recognized format would
    just be treated as strings, and the interpretation left to the custom
    plugin/client.

That works once things are configured, but how does a user configure
dimensions from the GUI or from REST config, which mind, are the
only supported configuration mechanisms? (you directly writing the files
is not a supported way, the on disk file format can and will change without
warning).

GUI wise, when you are configuring the custom dimensions, how do you
provide a list of valid values to the end user?
If you look at the DimensionEditor class you’ll see there is a check like the
following that enables/disables the possibility to configure dimensions;

GridCoverageReader reader = ((CoverageInfo) resource).getGridCoverageReader(null, null);
if(Number.class.isAssignableFrom(type)) {
String elev = reader.getMetadataValue(AbstractGridCoverage2DReader.HAS_ELEVATION_DOMAIN);
if(!Boolean.parseBoolean(elev)) {
disableDimension(type, configs, noAttributeMessage);
}
} else if(Date.class.isAssignableFrom(type)) {
String time = reader.getMetadataValue(AbstractGridCoverage2DReader.HAS_TIME_DOMAIN);
if(!Boolean.parseBoolean(time)) {
disableDimension(type, configs, noAttributeMessage);
}
}

The GUI must have a way to list the available dimensions, out of the canonical
ones, and allow the user to enable them.

An acceptable approach is to mimick the existing metadata keys:

/**

  • The time domain (comma separated list of values)
    */
    public static final String TIME_DOMAIN = “TIME_DOMAIN”;

/**

  • Time domain resolution (when using min/max/resolution)
    */
    public static final String TIME_DOMAIN_RESOLUTION = “TIME_DOMAIN_RESOLUTION”;

/**

  • If the time domain is available (or if a min/max/resolution approach has been chosen)
    */
    public static final String HAS_TIME_DOMAIN = “HAS_TIME_DOMAIN”;

/**

  • The time domain max value
    */
    public static final String TIME_DOMAIN_MAXIMUM = “TIME_DOMAIN_MAXIMUM”;

/**

  • The time domain min value
    */
    public static final String TIME_DOMAIN_MINIMUM = “TIME_DOMAIN_MINIMUM”;

For a custom dimensions we could have the following keys:
CUSTOM_DIMENSIONS (a comma separated list of the dimensions)
_TYPE (Date,Number,String?)

HAS__DOMAIN
_DOMAIN (the only solution for string dimensions)
_RESOLUTION (makes sense only for numeric/time)
_MINIMUM (makes sense only for numeric/time)
_MAXIMUM (makes sense only for numeric/time)

Around the above it should be possible to build all the configuration machinery and
also make meaningful choices when working against the requests coming down
(e.g., you give me a range request with resolution against a dimension with
string values and we can throw back a meaningful service exception)

Mind, whilst there is basically nothing left in core that does not have
a GUI I’m not saying that you have to build the GUI yourself, I’m saying
that whatever solution is chosen it has to allow the construction of a meaningful
GUI.

Speaking of requirements, the changes you are proposing are core ones,
they must be compounded with unit/integration
tests checking that the code is doing the right thing also with the custom
dimensions. See all the Dimensions*Test test classes here:
http://svn.codehaus.org/geoserver/trunk/src/wms/src/test/java/org/geoserver/wms/wms_1_1_1/
http://svn.codehaus.org/geoserver/trunk/src/wms/src/test/java/org/geoserver/wms/wms_1_3/

I don’t mean you have to build that many tests, but there must be some covering
custom dimensions handling. Which also means there must be a reader (even
a mocked up one) in the classpath that reports such dimensions.

If it sounds like a lot of work, consider that if you try to contribute
to the core code you also have to play by the same rules as a core developer.

  1. I believe the modified DefaultWebCoverageService100.getCoverage method
    can handle custom dimensions regardless of whether they are numeric. As it
    is now, the axis subset list items (created from the axisSubset elements in
    the request document) that are processed by this method are of type
    TypedLiteralType, which specifies all values as strings. The ELEVATION
    processing assumes that the values can be parsed to numbers, but I don’t see
    anything requiring the custom dimension processing to make such an
    assumption.

Read the WCS 1.0 spec quickly, the spec only cites numeric cases for
custom dimensions, but yeah, I don’t see it negating the possibility
for non numeric axes.

Cheers
Andrea

Ing. Andrea Aime
GeoSolutions S.A.S.
Tech lead

Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy

phone: +39 0584 962313
fax: +39 0584 962313
mob: +39 339 8844549

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.youtube.com/user/GeoSolutionsIT
http://www.linkedin.com/in/andreaaime
http://twitter.com/geowolf


geowolf wrote

That works once things are configured, but how does a user configure
dimensions from the GUI or from REST config, which mind, are the
only supported configuration mechanisms? (you directly writing the files
is not a supported way, the on disk file format can and will change
without
warning).

Can you clarify regarding the REST config for the coverage.xml file - I know
the user (i.e. coverage provider) does not directly write the file, but I
thought they directly specified the metadata entries as XML elements. In
which case they can simply use a dimensionInfo element - or lack thereof -
to indicate whether a particular metadata entry is to be treated as a
dimension. Or is there another way to specify metadata in the REST config
that I'm not aware of, which would require additional code changes to
support custom dimensions?

geowolf wrote

The GUI must have a way to list the available dimensions, out of the
canonical
ones, and allow the user to enable them.

An acceptable approach is to mimick the existing metadata keys:

/**
     * The time domain (comma separated list of values)
     */
    public static final String TIME_DOMAIN = "TIME_DOMAIN";

    /**
     * Time domain resolution (when using min/max/resolution)
     */
    public static final String TIME_DOMAIN_RESOLUTION =
"TIME_DOMAIN_RESOLUTION";

    /**
     * If the time domain is available (or if a min/max/resolution
approach
has been chosen)
     */
    public static final String HAS_TIME_DOMAIN = "HAS_TIME_DOMAIN";

    /**
     * The time domain max value
     */
    public static final String TIME_DOMAIN_MAXIMUM =
"TIME_DOMAIN_MAXIMUM";

    /**
     * The time domain min value
     */
    public static final String TIME_DOMAIN_MINIMUM =
"TIME_DOMAIN_MINIMUM";

For a custom dimensions we could have the following keys:
CUSTOM_DIMENSIONS (a comma separated list of the dimensions)
<MYDIMENSION>_TYPE (Date,Number,String?)

HAS_<MYDIMENSION>_DOMAIN
<MYDIMENSION>_DOMAIN (the only solution for string dimensions)
<MYDIMENSION>_RESOLUTION (makes sense only for numeric/time)
<MYDIMENSION>_MINIMUM (makes sense only for numeric/time)
<MYDIMENSION>_MAXIMUM (makes sense only for numeric/time)

I agree with this approach. Just to clarify: this amplifies but does not
conflict with the proposal described in my previous posts, correct?
So, besides developing the tests you described - and assuming I were not
inclined to build the GUI (which I appreciate your saying I would not
necessarily be responsible for!), can you tell me what the next step(s)
would be to getting this officially submitted?

Thanks,
Mike

--
View this message in context: http://osgeo-org.1560.n6.nabble.com/Modification-to-support-custom-dimensions-for-coverage-layers-tp4474094p4500433.html
Sent from the GeoServer - Dev mailing list archive at Nabble.com.