[Geoserver-devel] The importer build failures

GeoServer GeoServer Developer
1

Hi,
I’ve been looking at the importer build failures and think I’ve found out why.
The test I’ve been looking at is an import of a time based mosaic.

Here is how it goes,

  • the importer manually builds the index shapefile and the properties file for the mosaic
  • the mosaic reader looks at the directory, finds the mandatory sample_image file is missing,
    assumes the mosaic is uninitialized and overwrites both the property and the shapefile

Now… I guess the mosaic logic could be modified to just generate what’s missing,
but I don’t get why so much code was written to duplicate the usual mosaic behavior
when one just has to setup the property collector files and let the auto-generator work with it.

Also, the way the property file is written disables the usage of overviews (LevelsNum is hard coded to 1),
which means this import is only usable with toy datasets.

Any feedback?

(btw, there are two other failures, but they are both related to mosaics with time, so…)

If the issue cannot be solved very quickly, maybe we can disable mosaic support for the short
term, improve it, and re-enable it later.

Cheers
Andrea

–

== Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information ==

Ing. Andrea Aime

@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

2

Il 12/gen/2014 12:13 “Andrea Aime” <andrea.aime@anonymised.com> ha scritto:

Hi,
I’ve been looking at the importer build failures and think I’ve found out why.
The test I’ve been looking at is an import of a time based mosaic.

Here is how it goes,

  • the importer manually builds the index shapefile and the properties file for the mosaic

To use the generated shape file we could try to add the UseExistingSchema=true into the indexer.properties file which will ask to the mosaic engine to create a new mosaic looking at the shapefile instead of the directory, this should work for this example.

  • the mosaic reader looks at the directory, finds the mandatory sample_image file is missing,
    assumes the mosaic is uninitialized and overwrites both the property and the shapefile

Now… I guess the mosaic logic could be modified to just generate what’s missing,
but I don’t get why so much code was written to duplicate the usual mosaic behavior
when one just has to setup the property collector files and let the auto-generator work with it.

You are talking about the test code right?

Also, the way the property file is written disables the usage of overviews (LevelsNum is hard coded to 1),
which means this import is only usable with toy datasets.

Any feedback?

(btw, there are two other failures, but they are both related to mosaics with time, so…)

If the issue cannot be solved very quickly, maybe we can disable mosaic support for the short
term, improve it, and re-enable it later.

Cheers
Andrea

–
== Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information ==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today.
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk

Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

3

On Sun, Jan 12, 2014 at 1:47 PM, carlo cancellieri <
carlo.cancellieri@anonymised.com> wrote:

Il 12/gen/2014 12:13 "Andrea Aime" <andrea.aime@anonymised.com> ha
scritto:

>
> Hi,
> I've been looking at the importer build failures and think I've found
out why.
> The test I've been looking at is an import of a time based mosaic.
>
> Here is how it goes,
> * the importer manually builds the index shapefile and the properties
file for the mosaic

To use the generated shape file we could try to add the
UseExistingSchema=true into the indexer.properties file which will ask to
the mosaic engine to create a new mosaic looking at the shapefile instead
of the directory, this should work for this example.

Interesting, that might be useful. So the mosaic code would regenerate the
sample image only?
That one is mandatory, it's used by the mosaic code to figure out color and
sample model for the mosaic without the overhead
of opening a full granule.

> * the mosaic reader looks at the directory, finds the mandatory
sample_image file is missing,
> assumes the mosaic is uninitialized and overwrites both the property
and the shapefile
>
> Now.. I guess the mosaic logic could be modified to just generate what's
missing,
> but I don't get why so much code was written to duplicate the usual
mosaic behavior
> when one just has to setup the property collector files and let the
auto-generator work with it.

You are talking about the test code right?

Unfortunately no, it's the main importer code that duplicates the mosaic
internal machinery
(a completely independent implementation, that covers only part of the
functionality provided
by the original one).

Cheers
Andrea

--
== Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information ==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

4

Andrea,

Interesting, that might be useful. So the mosaic code would regenerate the sample image only?
That one is mandatory, it’s used by the mosaic code to figure out color and sample model for the mosaic without the overhead
of opening a full granule.

Yes, the mosaic engine will use the shapefile (or the store referred by an optional datastore.properties) to create the mosaic generating the sample_image and the MOSAIC.properties files. You may found some test cade about this feature using h2 db into the mosaic test code.
Note that this improvement is on 2.5.x don’t think that was ported on the 2.4.x.

Carlo

  • the mosaic reader looks at the directory, finds the mandatory sample_image file is missing,
    assumes the mosaic is uninitialized and overwrites both the property and the shapefile

Now… I guess the mosaic logic could be modified to just generate what’s missing,
but I don’t get why so much code was written to duplicate the usual mosaic behavior
when one just has to setup the property collector files and let the auto-generator work with it.

You are talking about the test code right?

Unfortunately no, it’s the main importer code that duplicates the mosaic internal machinery
(a completely independent implementation, that covers only part of the functionality provided
by the original one).

Cheers
Andrea

–
== Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information ==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

5

Yup, I implemented the mosaic support and that is code that I am not familiar with so not surprised there are issues with the implementation. I cobbled it together based on the docs and code samples i could find. Improvements very welcome, I’ll delegate to you and the mosaic experts as to what is appropriate.

···

On Sun, Jan 12, 2014 at 6:54 AM, carlo cancellieri <carlo.cancellieri@anonymised.com> wrote:

Andrea,

Interesting, that might be useful. So the mosaic code would regenerate the sample image only?
That one is mandatory, it’s used by the mosaic code to figure out color and sample model for the mosaic without the overhead
of opening a full granule.

Yes, the mosaic engine will use the shapefile (or the store referred by an optional datastore.properties) to create the mosaic generating the sample_image and the MOSAIC.properties files. You may found some test cade about this feature using h2 db into the mosaic test code.
Note that this improvement is on 2.5.x don’t think that was ported on the 2.4.x.

Carlo

  • the mosaic reader looks at the directory, finds the mandatory sample_image file is missing,
    assumes the mosaic is uninitialized and overwrites both the property and the shapefile

Now… I guess the mosaic logic could be modified to just generate what’s missing,
but I don’t get why so much code was written to duplicate the usual mosaic behavior
when one just has to setup the property collector files and let the auto-generator work with it.

You are talking about the test code right?

Unfortunately no, it’s the main importer code that duplicates the mosaic internal machinery
(a completely independent implementation, that covers only part of the functionality provided
by the original one).

Cheers
Andrea

–
== Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information ==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today.
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk

Geoserver-devel mailing list
Geoserver-devel@anonymised.comt
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

–

Justin Deoliveira
Vice President, Engineering | Boundless
jdeolive@anonymised.com
@j_deolive

6

On Mon, Jan 13, 2014 at 4:28 PM, Justin Deoliveira <
jdeolive@anonymised.com> wrote:

Yup, I implemented the mosaic support and that is code that I am not
familiar with so not surprised there are issues with the implementation. I
cobbled it together based on the docs and code samples i could find.
Improvements very welcome, I'll delegate to you and the mosaic experts as
to what is appropriate.

My instinct would be to strip the granule handling from the importer, have
it create the indexer.properties file instead,
and have the mosaic machinery build everything else automatically.

I'm just not sure whether we'd end up missing some feature that was
required in the importer, that the normal
mosaic machinery does not cover?

Cheers
Andrea

--
== Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information ==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

7

On Mon, Jan 13, 2014 at 8:42 AM, Andrea Aime
<andrea.aime@anonymised.com>wrote:

On Mon, Jan 13, 2014 at 4:28 PM, Justin Deoliveira <
jdeolive@anonymised.com> wrote:

Yup, I implemented the mosaic support and that is code that I am not
familiar with so not surprised there are issues with the implementation. I
cobbled it together based on the docs and code samples i could find.
Improvements very welcome, I'll delegate to you and the mosaic experts as
to what is appropriate.

My instinct would be to strip the granule handling from the importer, have
it create the indexer.properties file instead,
and have the mosaic machinery build everything else automatically.

That makes sense to me.

I'm just not sure whether we'd end up missing some feature that was
required in the importer, that the normal
mosaic machinery does not cover?

I think the biggest thing is that we want to preserve is that we want users
to be able to specify timestamps manually without having to use a regular
expression or date format string. This stems from mapstory in which users
upload time series data without a consistent file naming convention.

Cheers
Andrea

--
== Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information ==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

--
*Justin Deoliveira*
Vice President, Engineering | Boundless
jdeolive@anonymised.com
@j_deolive <https://twitter.com/j_deolive&gt;

8

On Mon, Jan 13, 2014 at 4:51 PM, Justin Deoliveira <
jdeolive@anonymised.com> wrote:

On Mon, Jan 13, 2014 at 8:42 AM, Andrea Aime <andrea.aime@anonymised.com
> wrote:

On Mon, Jan 13, 2014 at 4:28 PM, Justin Deoliveira <
jdeolive@anonymised.com> wrote:

Yup, I implemented the mosaic support and that is code that I am not
familiar with so not surprised there are issues with the implementation. I
cobbled it together based on the docs and code samples i could find.
Improvements very welcome, I'll delegate to you and the mosaic experts as
to what is appropriate.

My instinct would be to strip the granule handling from the importer,
have it create the indexer.properties file instead,
and have the mosaic machinery build everything else automatically.

That makes sense to me.

I'm just not sure whether we'd end up missing some feature that was
required in the importer, that the normal
mosaic machinery does not cover?

I think the biggest thing is that we want to preserve is that we want
users to be able to specify timestamps manually without having to use a
regular expression or date format string. This stems from mapstory in which
users upload time series data without a consistent file naming convention.

Ah, I see... wondering, and what if the files were renamed to include a iso
date at the end during import?
Would it be bad? Just checking what options we have, Carlo's suggestions
are good also, but what if we
wanted to allow the configuration of more property extractors... I mean,
we'd end up duplicating more and
more of the mosaic own functionality.

Cheers
Andrea

--
== Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information ==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

9

On Mon, Jan 13, 2014 at 8:56 AM, Andrea Aime
<andrea.aime@anonymised.com>wrote:

On Mon, Jan 13, 2014 at 4:51 PM, Justin Deoliveira <
jdeolive@anonymised.com> wrote:

On Mon, Jan 13, 2014 at 8:42 AM, Andrea Aime <
andrea.aime@anonymised.com> wrote:

On Mon, Jan 13, 2014 at 4:28 PM, Justin Deoliveira <
jdeolive@anonymised.com> wrote:

Yup, I implemented the mosaic support and that is code that I am not
familiar with so not surprised there are issues with the implementation. I
cobbled it together based on the docs and code samples i could find.
Improvements very welcome, I'll delegate to you and the mosaic experts as
to what is appropriate.

My instinct would be to strip the granule handling from the importer,
have it create the indexer.properties file instead,
and have the mosaic machinery build everything else automatically.

That makes sense to me.

I'm just not sure whether we'd end up missing some feature that was
required in the importer, that the normal
mosaic machinery does not cover?

I think the biggest thing is that we want to preserve is that we want
users to be able to specify timestamps manually without having to use a
regular expression or date format string. This stems from mapstory in which
users upload time series data without a consistent file naming convention.

Ah, I see... wondering, and what if the files were renamed to include a
iso date at the end during import?
Would it be bad? Just checking what options we have, Carlo's suggestions
are good also, but what if we
wanted to allow the configuration of more property extractors... I mean,
we'd end up duplicating more and
more of the mosaic own functionality.

CCing ian as he is more familiar with this use case. BUt my thought is yes
it would be fine to rename the files after the user has configured the
timestamps and proceeded with import.

Cheers
Andrea

--
== Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information ==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

--
*Justin Deoliveira*
Vice President, Engineering | Boundless
jdeolive@anonymised.com
@j_deolive <https://twitter.com/j_deolive&gt;

10

On Mon, Jan 13, 2014 at 8:42 AM, Andrea Aime <
andrea.aime@anonymised.com> wrote:

On Mon, Jan 13, 2014 at 4:28 PM, Justin Deoliveira <
jdeolive@anonymised.com> wrote:

Yup, I implemented the mosaic support and that is code that I am not
familiar with so not surprised there are issues with the implementation. I
cobbled it together based on the docs and code samples i could find.
Improvements very welcome, I'll delegate to you and the mosaic experts as
to what is appropriate.

My instinct would be to strip the granule handling from the importer,
have it create the indexer.properties file instead,
and have the mosaic machinery build everything else automatically.

That makes sense to me.

I'm just not sure whether we'd end up missing some feature that was
required in the importer, that the normal
mosaic machinery does not cover?

I think the biggest thing is that we want to preserve is that we want
users to be able to specify timestamps manually without having to use a
regular expression or date format string. This stems from mapstory in which
users upload time series data without a consistent file naming convention.

The goals were 3, really:

1) don't make end users have to specify a regular expression (since they'd
probably rather give up)
2) provide reasonable support for finding ISO dates in the file names
without any configuration
3) allow specifying a format or literal value if some ambiguity or
inconsistency existed

Personally, I'm happy with 1 and 2 and in the case of 3 just saying - "your
files should adhere to one of the following conventions".

Ah, I see... wondering, and what if the files were renamed to include a
iso date at the end during import?
Would it be bad? Just checking what options we have, Carlo's suggestions
are good also, but what if we
wanted to allow the configuration of more property extractors... I mean,
we'd end up duplicating more and
more of the mosaic own functionality.

CCing ian as he is more familiar with this use case. BUt my thought is yes
it would be fine to rename the files after the user has configured the
timestamps and proceeded with import.

I can't comment much on the proposed implementation/internals except to say
that renaming files would be fine, but if it's possible to support some
reasonable defaults and (if possible) also allow a fallback to supporting
user-specified values (sans regex), that would fulfill the use case.

Thanks for looking into the failures Andrea, not sure why I wasn't seeing
them...

--
Ian Schneider
Software Engineer | Boundless
ischneider@anonymised.com

11

On Tue, Jan 14, 2014 at 9:20 PM, Ian Schneider
<ischneider@anonymised.com>wrote:

On Mon, Jan 13, 2014 at 8:42 AM, Andrea Aime <
andrea.aime@anonymised.com> wrote:

On Mon, Jan 13, 2014 at 4:28 PM, Justin Deoliveira <
jdeolive@anonymised.com> wrote:

Yup, I implemented the mosaic support and that is code that I am not
familiar with so not surprised there are issues with the implementation. I
cobbled it together based on the docs and code samples i could find.
Improvements very welcome, I'll delegate to you and the mosaic experts as
to what is appropriate.

My instinct would be to strip the granule handling from the importer,
have it create the indexer.properties file instead,
and have the mosaic machinery build everything else automatically.

That makes sense to me.

I'm just not sure whether we'd end up missing some feature that was
required in the importer, that the normal
mosaic machinery does not cover?

I think the biggest thing is that we want to preserve is that we want
users to be able to specify timestamps manually without having to use a
regular expression or date format string. This stems from mapstory in which
users upload time series data without a consistent file naming convention.

The goals were 3, really:

1) don't make end users have to specify a regular expression (since they'd
probably rather give up)
2) provide reasonable support for finding ISO dates in the file names
without any configuration
3) allow specifying a format or literal value if some ambiguity or
inconsistency existed

Personally, I'm happy with 1 and 2 and in the case of 3 just saying -
"your files should adhere to one of the following conventions".

Ah, I see... wondering, and what if the files were renamed to include a
iso date at the end during import?
Would it be bad? Just checking what options we have, Carlo's suggestions
are good also, but what if we
wanted to allow the configuration of more property extractors... I mean,
we'd end up duplicating more and
more of the mosaic own functionality.

CCing ian as he is more familiar with this use case. BUt my thought is
yes it would be fine to rename the files after the user has configured the
timestamps and proceeded with import.

I can't comment much on the proposed implementation/internals except to
say that renaming files would be fine, but if it's possible to support some
reasonable defaults and (if possible) also allow a fallback to supporting
user-specified values (sans regex), that would fulfill the use case.

I see, thanks for the feedback.

Given that the beta release is in just a few days, do you mind if we
disable the mosaic import functionality for the moment,
in order to graduate the module, and re-enable it once it can be fixed?

Thanks for looking into the failures Andrea, not sure why I wasn't seeing
them...

I guess you had out of date mosaic jars in your classpath, it happens if
you don't also build geotools from sources

Cheers
Andrea

--
== Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information ==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

12

I see, thanks for the feedback.

Given that the beta release is in just a few days, do you mind if we
disable the mosaic import functionality for the moment,
in order to graduate the module, and re-enable it once it can be fixed?

Disabling is fine with me.

Thanks for looking into the failures Andrea, not sure why I wasn't seeing
them...

I guess you had out of date mosaic jars in your classpath, it happens if
you don't also build geotools from sources

Makes sense that could have happened.

--
Ian Schneider
Software Engineer | Boundless
ischneider@anonymised.com

13

On Tue, Jan 14, 2014 at 2:31 PM, Ian Schneider
<ischneider@anonymised.com>wrote:

I see, thanks for the feedback.

Given that the beta release is in just a few days, do you mind if we
disable the mosaic import functionality for the moment,
in order to graduate the module, and re-enable it once it can be fixed?

Disabling is fine with me.

Hmmm... and when will that be? Disabling functionality in order to promote
a module seems a bit extreme to me. Unless there is an actual mandate to
fix these issues soon I personally would put off moving to extension.

Thanks for looking into the failures Andrea, not sure why I wasn't
seeing them...

I guess you had out of date mosaic jars in your classpath, it happens if
you don't also build geotools from sources

Makes sense that could have happened.

--
Ian Schneider
Software Engineer | Boundless
ischneider@anonymised.com

--
*Justin Deoliveira*
Vice President, Engineering | Boundless
jdeolive@anonymised.com
@j_deolive <https://twitter.com/j_deolive&gt;