[Geoserver-users] New image mosaic does not work in WCS 2.0.1

Hi,

I had removed that old installation so I made a new test with Geoserver 2.8-beta and most probably with different ortophotos as well. Fortunately (?) I could repeat the error. The four GeoTIFFs I have are located side by side and the gross BBOX is a rectangle of 12 km by 12 km. The coverage works perfectly in the layer preview. The reason for the NPE with GetCoverage is almost identical. Obviously there has been some small code change in GetCoverage.java.

at org.geoserver.wcs2_0.GetCoverage.readCoverage(GetCoverage.java:761)
at org.geoserver.wcs2_0.GetCoverage.setupCoverage(GetCoverage.java:335) #was line 338
at org.geoserver.wcs2_0.GetCoverage.run(GetCoverage.java:286) #was line 289

I must try to make a new test either with images which can be downloaded from the web or by decreasing the size of my test images because now they are 420 MB each.

-Jukka-

Andrea Aime wrote:

···

On Fri, Sep 25, 2015 at 10:05 AM, Andrea Aime <andrea.aime@…1107…> wrote:

Hi Jukka,

indeed it seems the read for the coverage failed and returned null (no data in the requested area).

Do you have some sample data for your case that I can try out?

Btw, looking at the code, it could also be this case:

https://osgeo-org.atlassian.net/browse/GEOS-7211

Cheers

Andrea

==

GeoServer Professional Services from the experts! Visit

http://goo.gl/it488V for more information.

==

Ing. Andrea Aime

@geowolf

Technical Lead

GeoSolutions S.A.S.

Via Poggio alle Viti 1187

55054 Massarosa (LU)

Italy

phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it

http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


Hi,

Here is a procedure for creating the issue:

Download 4 ortophotos:

http://kartat.kapsi.fi/files/orto/etrs-tm35fin/mavi_v_25000_50/2015/L44/02m/1/L4422A.jp2

http://kartat.kapsi.fi/files/orto/etrs-tm35fin/mavi_v_25000_50/2015/L44/02m/1/L4422B.jp2

http://kartat.kapsi.fi/files/orto/etrs-tm35fin/mavi_v_25000_50/2015/L44/02m/1/L4422C.jp2

http://kartat.kapsi.fi/files/orto/etrs-tm35fin/mavi_v_25000_50/2015/L44/02m/1/L4422D.jp2

Convert JPEG2000 files into GeoTIFFs
gdal_translate -of GTiff -co tiled=yes L4422A.jp2 L4422A.tif
gdal_translate -of GTiff -co tiled=yes L4422B.jp2 L4422B.tif
gdal_translate -of GTiff -co tiled=yes L4422C.jp2 L4422C.tif
gdal_translate -of GTiff -co tiled=yes L4422D.jp2 L4422D.tif

Create store and publish the layer and send GetCoverage:

http://localhost:8080/geoserver/wcs?service=WCS&version=2.0.1&request=describecoverage&coverageid=topp__orto_nls

DescribeCoverage from this layer is attached. There are some odd things in the document:

  1. The Envelope in boundedBy has coordinates in wrong order N-E though they should be E-N

gml:boundedBy

<gml:Envelope srsName=“http://www.opengis.net/def/crs/EPSG/0/3067” axisLabels=“E N” uomLabels=“m m” srsDimension=“2”>

gml:lowerCorner6750000.0 404000.0</gml:lowerCorner>

gml:upperCorner6762000.0 416000.0</gml:upperCorner>

</gml:Envelope>

</gml:boundedBy>

  1. If may be correct to use the axisOrder rule +2 +1 but other WCS servers which I have tried do not use it (MapServer, Rasdaman)

gml:GridFunction

<gml:sequenceRule axisOrder=“+2 +1”>Linear</gml:sequenceRule>

gml:startPoint0 0</gml:startPoint>

</gml:GridFunction>

  1. It is too difficult for me to follow it the mentioned GridFunction together with these origin and offsetVectors lead to a correct result or not

gml:origin

<gml:Point gml:id=“p00_topp__orto_nls” srsName=“http://www.opengis.net/def/crs/EPSG/0/3067”>

gml:pos6761999.75 404000.25</gml:pos>

</gml:Point>

</gml:origin>

<gml:offsetVector srsName=“http://www.opengis.net/def/crs/EPSG/0/3067”>0.0 0.5</gml:offsetVector>

<gml:offsetVector srsName=“http://www.opengis.net/def/crs/EPSG/0/3067”>-0.5 0.0</gml:offsetVector>

So, first vector is increasing 404000.25 (which is Easting) in 0.5 m steps, the other is decreasing 6761999.75 (which is Northing in 0.5 m steps and at some moment the whole coverage is turned around with the GridFunction.

I attach also WCS 2.0.1 GetCoverage from Mapserver 7.0 so it is possible to compare how it is expressing boundedBy, origin, and offSet vectors. At least for me this GML without GridFunction is easier to understand. Rasdaman is creating GML pretty much alike MapServer.

There seems to be once again half a pixel shifts in the origins from Geoserver vs. Mapserver.

-Jukka Rahkonen-

WCS_DescribeCoverage.txt (4.21 KB)

WCS_DescribeCoverage_Mapserver.txt (3.23 KB)

···

Rahkonen Jukka (MML) wrote:

Hi,

I had removed that old installation so I made a new test with Geoserver 2.8-beta and most probably with different ortophotos as well. Fortunately (?) I could repeat the error. The four GeoTIFFs I have are located side by side and the gross BBOX is a rectangle of 12 km by 12 km. The coverage works perfectly in the layer preview. The reason for the NPE with GetCoverage is almost identical. Obviously there has been some small code change in GetCoverage.java.

at org.geoserver.wcs2_0.GetCoverage.readCoverage(GetCoverage.java:761)
at org.geoserver.wcs2_0.GetCoverage.setupCoverage(GetCoverage.java:335) #was line 338
at org.geoserver.wcs2_0.GetCoverage.run(GetCoverage.java:286) #was line 289

I must try to make a new test either with images which can be downloaded from the web or by decreasing the size of my test images because now they are 420 MB each.

-Jukka-

Andrea Aime wrote:

On Fri, Sep 25, 2015 at 10:05 AM, Andrea Aime <andrea.aime@anonymised.com07…> wrote:

Hi Jukka,

indeed it seems the read for the coverage failed and returned null (no data in the requested area).

Do you have some sample data for your case that I can try out?

Btw, looking at the code, it could also be this case:

https://osgeo-org.atlassian.net/browse/GEOS-7211

Cheers

Andrea

==

GeoServer Professional Services from the experts! Visit

http://goo.gl/it488V for more information.

==

Ing. Andrea Aime

@geowolf

Technical Lead

GeoSolutions S.A.S.

Via Poggio alle Viti 1187

55054 Massarosa (LU)

Italy

phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39 339 8844549

http://www.geo-solutions.it

http://twitter.com/geosolutions_it

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy’s New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


On Fri, Sep 25, 2015 at 1:39 PM, Rahkonen Jukka (MML) <
jukka.rahkonen@anonymised.com> wrote:

Hi,

Here is a procedure for creating the issue:

Download 4 ortophotos:

http://kartat.kapsi.fi/files/orto/etrs-tm35fin/mavi_v_25000_50/2015/L44/02m/1/L4422A.jp2

http://kartat.kapsi.fi/files/orto/etrs-tm35fin/mavi_v_25000_50/2015/L44/02m/1/L4422B.jp2

http://kartat.kapsi.fi/files/orto/etrs-tm35fin/mavi_v_25000_50/2015/L44/02m/1/L4422C.jp2

http://kartat.kapsi.fi/files/orto/etrs-tm35fin/mavi_v_25000_50/2015/L44/02m/1/L4422D.jp2

Convert JPEG2000 files into GeoTIFFs
gdal_translate -of GTiff -co tiled=yes L4422A.jp2 L4422A.tif
gdal_translate -of GTiff -co tiled=yes L4422B.jp2 L4422B.tif
gdal_translate -of GTiff -co tiled=yes L4422C.jp2 L4422C.tif
gdal_translate -of GTiff -co tiled=yes L4422D.jp2 L4422D.tif

Wow, they really put my machine to the test, gdal was using 6GB+ of memory
during
the translation (no kakadu).

Create store and publish the layer and send GetCoverage:

http://localhost:8080/geoserver/wcs?service=WCS&version=2.0.1&request=describecoverage&coverageid=topp__orto_nls

DescribeCoverage from this layer is attached. There are some odd things in
the document:

1) The Envelope in boundedBy has coordinates in wrong order N-E though
they should be E-N

<gml:boundedBy>

<gml:Envelope srsName="http://www.opengis.net/def/crs/EPSG/0/3067&quot;
axisLabels="E N" uomLabels="m m" srsDimension="2">

<gml:lowerCorner>6750000.0 404000.0</gml:lowerCorner>

<gml:upperCorner>6762000.0 416000.0</gml:upperCorner>

</gml:Envelope>

</gml:boundedBy>

Fixed as part of https://osgeo-org.atlassian.net/browse/GEOS-7142, I now
get:

gml:Envelope srsName="http://www.opengis.net/def/crs/EPSG/0/3067&quot; axisLabels
="E N" uomLabels="m m" srsDimension="2">
<gml:lowerCorner>404000.0 6750000.0</gml:lowerCorner>
<gml:upperCorner>416000.0 6762000.0</gml:upperCorner>
</gml:Envelope>
</gml:boundedBy>

Full describe results attached.

2) If may be correct to use the axisOrder rule +2 +1 but other WCS servers
which I have tried do not use it (MapServer, Rasdaman)

<gml:GridFunction>

<gml:sequenceRule axisOrder="+2 +1">Linear</gml:sequenceRule>

<gml:startPoint>0 0</gml:startPoint>

</gml:GridFunction>

This is used to indicate the grid pixels have one orientation, but the axis
order is the opposite.
What do MapServer/Rasdaman do to describe a coverage in
http://www.opengis.net/def/crs/EPSG/0/4326, which has flipped axis in WCS
2.0?

3) It is too difficult for me to follow it the mentioned GridFunction
together with these origin and offsetVectors lead to a correct result or not

<gml:origin>

<gml:Point gml:id="p00_topp__orto_nls" srsName="
http://www.opengis.net/def/crs/EPSG/0/3067&quot;&gt;

<gml:pos>6761999.75 404000.25</gml:pos>

</gml:Point>

</gml:origin>

<gml:offsetVector srsName="http://www.opengis.net/def/crs/EPSG/0/3067&quot;&gt;0\.0
0.5</gml:offsetVector>

<gml:offsetVector srsName="http://www.opengis.net/def/crs/EPSG/0/3067&quot;&gt;\-0\.5
0.0</gml:offsetVector>

So, first vector is increasing 404000.25 (which is Easting) in 0.5 m
steps, the other is decreasing 6761999.75 (which is Northing in 0.5 m steps
and at some moment the whole coverage is turned around with the
GridFunction.

I attach also WCS 2.0.1 GetCoverage from Mapserver 7.0 so it is possible
to compare how it is expressing boundedBy, origin, and offSet vectors. At
least for me this GML without GridFunction is easier to understand.
Rasdaman is creating GML pretty much alike MapServer.

There seems to be once again half a pixel shifts in the origins from
Geoserver vs. Mapserver.

GeoServer is as far as I know trying to repect the OGC rule of "pixel is
center", which is not how raster data is saved
unfortunately (normally raster data is encoded as pixel is corner instead).

I've then tried running:

http://localhost:8080/geoserver/wcs?service=WCS&version=2.0.0&request=GetCoverage&coverageid=finmosaic

and now GeoServer is busy writing the result to a temp file (we cannot
stream it directly), at least by the looks
of the following trace (which I got using jstack, it's not a recorded
error, it's a way to check what a Java app
is busy working on):

"btpool0-2" prio=10 tid=0x00007f0930006800 nid=0x1c6a runnable
[0x00007f09743ab000]
   java.lang.Thread.State: RUNNABLE
        at java.io.RandomAccessFile.writeBytes0(Native Method)
        at java.io.RandomAccessFile.writeBytes(RandomAccessFile.java:520)
        at java.io.RandomAccessFile.write(RandomAccessFile.java:550)
        at
javax.imageio.stream.FileCacheImageOutputStream.write(FileCacheImageOutputStream.java:141)
        at
it.geosolutions.imageioimpl.plugins.tiff.TIFFNullCompressor.encode(TIFFNullCompressor.java:104)
        at
it.geosolutions.imageioimpl.plugins.tiff.TIFFImageWriter.writeTile(TIFFImageWriter.java:1905)
        at
it.geosolutions.imageioimpl.plugins.tiff.TIFFImageWriter.write(TIFFImageWriter.java:2920)
        at
it.geosolutions.imageioimpl.plugins.tiff.TIFFImageWriter.write(TIFFImageWriter.java:2614)
        at
org.geotools.gce.geotiff.GeoTiffWriter.writeImage(GeoTiffWriter.java:427)
        at
org.geotools.gce.geotiff.GeoTiffWriter.write(GeoTiffWriter.java:271)

This is going to take a while, the mosaic is 24k by 24k... looking at it,
we probably need some tweaks in the
code to generate such a large output, I can see there are extra operations
in the chain that could be
avoided for this case, and we could force a "reasonable" tiling structure
in the output too, now it's
following the input one, which is... really bad.

Here is a prelim gdalinfo against the temp file, to give you an idea:

gdalinfo /tmp/imageio3762159434441867746.tmp
Driver: GTiff/GeoTIFF
Files: /tmp/imageio3762159434441867746.tmp
Size is 24000, 24000
Coordinate System is:
PROJCS["ETRS89 / TM35FIN(E,N)",
    GEOGCS["ETRS89",
        DATUM["European_Terrestrial_Reference_System_1989",
            SPHEROID["GRS 1980",6378137,298.2572221010002,
                AUTHORITY["EPSG","7019"]],
            AUTHORITY["EPSG","6258"]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433],
        AUTHORITY["EPSG","4258"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",27],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AUTHORITY["EPSG","3067"]]
Origin = (404000.000000000000000,6762000.000000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Metadata:
  AREA_OR_POINT=Area
  TIFFTAG_RESOLUTIONUNIT=1 (unitless)
  TIFFTAG_XRESOLUTION=1
  TIFFTAG_YRESOLUTION=1
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 404000.000, 6762000.000) ( 25d13'33.38"E, 60d58'52.54"N)
Lower Left ( 404000.000, 6750000.000) ( 25d13'54.90"E, 60d52'24.88"N)
Upper Right ( 416000.000, 6762000.000) ( 25d26'51.34"E, 60d59' 2.39"N)
Lower Right ( 416000.000, 6750000.000) ( 25d27'10.18"E, 60d52'34.68"N)
Center ( 410000.000, 6756000.000) ( 25d20'22.45"E, 60d55'43.79"N)
Band 1 Block=24000x8 Type=Byte, ColorInterp=Red
  NoData Value=0
Band 2 Block=24000x8 Type=Byte, ColorInterp=Green
  NoData Value=0
Band 3 Block=24000x8 Type=Byte, ColorInterp=Blue
  NoData Value=0

Cheers
Andrea

--

GeoServer Professional Services from the experts! Visit
http://goo.gl/it488V for more information.

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
mob: +39 339 8844549

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

*AVVERTENZE AI SENSI DEL D.Lgs. 196/2003*

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.

-------------------------------------------------------

describe.xml (4.13 KB)