[Geoserver-devel] Image pyramid improvements and tutorial

Stephen_Mather2 · March 23, 2010, 2:00pm

I should say, the file in question is called retile_test1.zip.

Steve

-----Original Message-----
From: Stephen V. Mather [mailto:svm@anonymised.com]
Sent: Monday, March 22, 2010 5:11 PM
To: 'Simone Giannecchini'
Cc: 'geoserver-devel@lists.sourceforge.net'
Subject: RE: [Geoserver-devel] Image pyramid improvements and tutorial

The test data are a little bigger than expected-- 195MB. They will be
available shortly at the following FTP site:

ftp://ftp.clevelandmetroparks.com/pdnr/out

Thanks,
Steve

Stephen Mather, GIS Manager
Cleveland Metroparks
4101 Fulton Pkwy
Cleveland, OH 44144
svm@anonymised.com
Phone: (216) 635-3243
FAX: (216) 635-3286

-----Original Message-----
From: simboss1@anonymised.com [mailto:simboss1@anonymised.com] On Behalf Of Simone
Giannecchini
Sent: Saturday, March 20, 2010 10:16 AM
To: svm@anonymised.com
Cc: geoserver-devel@lists.sourceforge.net
Subject: Re: [Geoserver-devel] Image pyramid improvements and tutorial

I guess that doing some testing with your data could be helpful. I have some
uncommitted fixes/improvements that might solve these issues.

Simone

On 3/19/10, Stephen V. Mather <svm@anonymised.com> wrote:

Qcuik questions:
- are you adding a reprojection in the mix while you are doing this
tests?

I have tried it both with and without reprojection, to the same
effect.

- are all your granules of the same resolution?

Yes, they are all 0.15m pixels in 5000x4000 images. I have tried both
leaving the existing 0 level images at 5000x4000 (-pyramidOnly) and
retiling to 2048x2048, both with the same result.

Steve

On Tue, Mar 16, 2010 at 1:13 PM, Stephen Mather
<svm@anonymised.com> wrote:

Hi Andrea,
     I've been working on giving it a spin with our own data. The
tutorial is excellent. Easy to read and follow.
     I'm currently testing it on a Windows 2003 server with nightly
build geoserver-2.0.x-2010-03-03-bin.zip, and
geoserver-2.0.2-SNAPSHOT-pyramid-plugin.zip
     The test I've been doing is with 0.5ft (0.15m) color orthophotos
that sum to about 150GB as TIFFs. They are currently tiled as
5000x4000 images. I've retiled them with gdal_retile as 2048x2048
images with overviews with the following command:

gdal_retile -v -s_srs "EPSG:102722" -levels 5 -ps 2048 2048
-targetdir test3 --optfile tilelist.txt

I'm currently working on a subset of the data for testing. Checking
all the tiles generated from gdal_retile, they all look fine in an
image viewer, and check out (with a spot check) with gdalinfo.

When I load them in GeoServer, I get a strange effect. Some of the
tiles never render at their full resolution for a given zoom level,
but at a reduced resolution. If I load ~16 tiles vs. loading e.g.
40+ tiles the places where this happens shifts, i.e. tiles that were
a problem before stop being a problem, while other tiles exhibit the
same behavior.

Does this make sense, is this an effect you've seen, and would images
and log files be helpful?

Thanks,
Steve

Andrea Aime-4 wrote:

Hi all,
lately I've been working, under OpenGeo request, to improve the
situation vs image pyramids, in particular, making it easier to
configure a file based one.

...

Is anyone interested to give the tutorial a spin, a review? Is there
anything missing or that could be done better?

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

--------------------------------------------------------------------
-
---------
SOLARIS 10 is the OS for Data Centers - provides features such as
DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris
10 NOW http://p.sf.net/sfu/solaris-dev2dev
_______________________________________________
Geoserver-devel mailing list Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
View this message in context:

http://old.nabble.com/Image-pyramid-improvements-and-tutorial-tp276086
54p279
10740.html

Sent from the GeoServer - Dev mailing list archive at Nabble.com.

---------------------------------------------------------------------
-
--------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance. See
why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Geoserver-devel mailing list Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

----------------------------------------------------------------------
--------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance. See
why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Geoserver-devel mailing list Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
-------------------------------------------------------
Ing. Simone Giannecchini
GeoSolutions S.A.S.
Founder - Software Engineer
Via Carignoni 51
55041 Camaiore (LU)
Italy

phone: +39 0584983027
fax: +39 0584983027
mob: +39 333 8128928

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.linkedin.com/in/simonegiannecchini
http://twitter.com/simogeo

-------------------------------------------------------

Andrea_Aime3 · March 23, 2010, 2:47pm

Stephen V. Mather ha scritto:

I should say, the file in question is called retile_test1.zip.

I looked into this file briefly and found the issue, the files at the
lowest level have different pixel resolutions.

I can be seen by running on a Linux/Unix command line the following
command, which extracts the pixel size from the gdalinfo output
on each of the tiff files in the root directory.

find . -name "*.tif" -exec gdalinfo {} \; | grep "Pixel Size"

The result is:

Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (4.000000000000000,-4.000000000000000)
Pixel Size = (8.000000000000000,-8.000000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (2.000000000000000,-2.000000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (16.000000000000000,-16.000000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)

As you can see there's a hodge podge of different resolutions,
I'm not surprised the pyramid plugin is not working.

How did you get this mixture?

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

Stephen_Mather2 · March 23, 2010, 3:28pm

Hi Andrea,
I sent the whole tree of data, rather than just the raw data. If I
check my 0 level tier for pixel size, here's what I get (please excuse the
DOS, I do so miss Linux and System 5 Unix)

FOR %f IN (*.tif) DO gdalinfo %f | grep Pixel >> temp.txt
more temp.txt

Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)

If I instead decend the file hierarchy, like find does:

for /R %f in (*.tif) do gdalinfo %f | grep Pixel >> temp1.txt
more temp1.txt

Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Pixel Size = (2.000000000000000,-2.000000000000000)
Pixel Size = (2.000000000000000,-2.000000000000000)
Pixel Size = (4.000000000000000,-4.000000000000000)
Pixel Size = (8.000000000000000,-8.000000000000000)
Pixel Size = (16.000000000000000,-16.000000000000000)

I get a range of pixel sizes from my 0 level tier at 0.5 feet to my 5th
level tier at 16 feet.

This is the result I expect when I run gdal_retile. Is this not what the
extension expects to see?

Thanks,
Steve

Stephen Mather, GIS Manager
Cleveland Metroparks
4101 Fulton Pkwy
Cleveland, OH 44144
svm@anonymised.com
Phone: (216) 635-3243
FAX: (216) 635-3286

-----Original Message-----
From: Andrea Aime [mailto:aaime@anonymised.com]
Sent: Tuesday, March 23, 2010 10:48 AM
To: svm@anonymised.com
Cc: 'Simone Giannecchini'; geoserver-devel@lists.sourceforge.net
Subject: Re: [Geoserver-devel] Image pyramid improvements and tutorial

Stephen V. Mather ha scritto:

I should say, the file in question is called retile_test1.zip.

I looked into this file briefly and found the issue, the files at the lowest
level have different pixel resolutions.

I can be seen by running on a Linux/Unix command line the following command,
which extracts the pixel size from the gdalinfo output on each of the tiff
files in the root directory.

find . -name "*.tif" -exec gdalinfo {} \; | grep "Pixel Size"

The result is:

Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (4.000000000000000,-4.000000000000000)
Pixel Size = (8.000000000000000,-8.000000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (2.000000000000000,-2.000000000000000)
Pixel Size = (2.000000000000000,-2.000000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (16.000000000000000,-16.000000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)

As you can see there's a hodge podge of different resolutions, I'm not
surprised the pyramid plugin is not working.

How did you get this mixture?

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

Andrea_Aime3 · March 23, 2010, 3:46pm

Stephen V. Mather ha scritto:

Hi Andrea,
I sent the whole tree of data, rather than just the raw data. If I
check my 0 level tier for pixel size, here's what I get (please excuse the
DOS, I do so miss Linux and System 5 Unix)

FOR %f IN (*.tif) DO gdalinfo %f | grep Pixel >> temp.txt
more temp.txt

Pixel Size = (0.500000000000000,-0.500000000000000)

If I instead decend the file hierarchy, like find does:

Arg, my brain was not completely connected when I run
that command, forgot to disable recursion in find...

sorry, I guess I'll need to have another look but right
now I don't have time to debug it, but I'll keep in mind
it's there.
Someone else might have time to look into it before me.

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

Simone_Giannecchini3 · March 24, 2010, 9:30am

Ciao Stephen,
I downloaded the data and ran a quick test.
The pyramid creation was smooth, although I noticed that the geotiff
files you have provided us with do not contain a proper EPSG code for
the CRS.
I picked one that was close enough and forced a reprojection (not the
best thing performance wise). Things worked for me in the end
(although as I said I have a few local changes that might have helped)

My questions for you are as follows:

- do you know a valid EPSG code for this dataset?
- why you using tiff that big for the single granule/tiles?

Ciao,
Simone.
-------------------------------------------------------
Ing. Simone Giannecchini
GeoSolutions S.A.S.
Founder - Software Engineer
Via Carignoni 51
55041 Camaiore (LU)
Italy

phone: +39 0584983027
fax: +39 0584983027
mob: +39 333 8128928

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.linkedin.com/in/simonegiannecchini
http://twitter.com/simogeo

-------------------------------------------------------

On Tue, Mar 23, 2010 at 3:00 PM, Stephen V. Mather
<svm@anonymised.com> wrote:

I should say, the file in question is called retile_test1.zip.

Steve

-----Original Message-----
From: Stephen V. Mather [mailto:svm@anonymised.com]
Sent: Monday, March 22, 2010 5:11 PM
To: 'Simone Giannecchini'
Cc: 'geoserver-devel@lists.sourceforge.net'
Subject: RE: [Geoserver-devel] Image pyramid improvements and tutorial

The test data are a little bigger than expected-- 195MB. They will be
available shortly at the following FTP site:

ftp://ftp.clevelandmetroparks.com/pdnr/out

Thanks,
Steve

Stephen Mather, GIS Manager
Cleveland Metroparks
4101 Fulton Pkwy
Cleveland, OH 44144
svm@anonymised.com
Phone: (216) 635-3243
FAX: (216) 635-3286

-----Original Message-----
From: simboss1@anonymised.com [mailto:simboss1@anonymised.com] On Behalf Of Simone
Giannecchini
Sent: Saturday, March 20, 2010 10:16 AM
To: svm@anonymised.com
Cc: geoserver-devel@lists.sourceforge.net
Subject: Re: [Geoserver-devel] Image pyramid improvements and tutorial

I guess that doing some testing with your data could be helpful. I have some
uncommitted fixes/improvements that might solve these issues.

Simone

On 3/19/10, Stephen V. Mather <svm@anonymised.com> wrote:

Qcuik questions:
- are you adding a reprojection in the mix while you are doing this
tests?

I have tried it both with and without reprojection, to the same
effect.

- are all your granules of the same resolution?

Yes, they are all 0.15m pixels in 5000x4000 images. I have tried both
leaving the existing 0 level images at 5000x4000 (-pyramidOnly) and
retiling to 2048x2048, both with the same result.

Steve

On Tue, Mar 16, 2010 at 1:13 PM, Stephen Mather
<svm@anonymised.com> wrote:

Hi Andrea,
I've been working on giving it a spin with our own data. The
tutorial is excellent. Easy to read and follow.
I'm currently testing it on a Windows 2003 server with nightly
build geoserver-2.0.x-2010-03-03-bin.zip, and
geoserver-2.0.2-SNAPSHOT-pyramid-plugin.zip
The test I've been doing is with 0.5ft (0.15m) color orthophotos
that sum to about 150GB as TIFFs. They are currently tiled as
5000x4000 images. I've retiled them with gdal_retile as 2048x2048
images with overviews with the following command:

gdal_retile -v -s_srs "EPSG:102722" -levels 5 -ps 2048 2048
-targetdir test3 --optfile tilelist.txt

I'm currently working on a subset of the data for testing. Checking
all the tiles generated from gdal_retile, they all look fine in an
image viewer, and check out (with a spot check) with gdalinfo.

When I load them in GeoServer, I get a strange effect. Some of the
tiles never render at their full resolution for a given zoom level,
but at a reduced resolution. If I load ~16 tiles vs. loading e.g.
40+ tiles the places where this happens shifts, i.e. tiles that were
a problem before stop being a problem, while other tiles exhibit the
same behavior.

Does this make sense, is this an effect you've seen, and would images
and log files be helpful?

Thanks,
Steve

Andrea Aime-4 wrote:

Hi all,
lately I've been working, under OpenGeo request, to improve the
situation vs image pyramids, in particular, making it easier to
configure a file based one.

...

Is anyone interested to give the tutorial a spin, a review? Is there
anything missing or that could be done better?

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

--------------------------------------------------------------------
-
---------
SOLARIS 10 is the OS for Data Centers - provides features such as
DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris
10 NOW http://p.sf.net/sfu/solaris-dev2dev
_______________________________________________
Geoserver-devel mailing list Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
View this message in context:

http://old.nabble.com/Image-pyramid-improvements-and-tutorial-tp276086
54p279
10740.html

Sent from the GeoServer - Dev mailing list archive at Nabble.com.

---------------------------------------------------------------------
-
--------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance. See
why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Geoserver-devel mailing list Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

----------------------------------------------------------------------
--------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance. See
why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Geoserver-devel mailing list Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
-------------------------------------------------------
Ing. Simone Giannecchini
GeoSolutions S.A.S.
Founder - Software Engineer
Via Carignoni 51
55041 Camaiore (LU)
Italy

phone: +39 0584983027
fax: +39 0584983027
mob: +39 333 8128928

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.linkedin.com/in/simonegiannecchini
http://twitter.com/simogeo

-------------------------------------------------------

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Stephen_Mather2 · March 24, 2010, 1:01pm

Hi Simone,
The correct CRS is EPSG:3728,
http://spatialreference.org/ref/epsg/3728/, although EPSG:102722 is pretty
close.
The size of the tiff is, well, either cheapness or laziness. The
current format, naming convention and size is what has been used by our
engineers in CAD for 8 years, and is referenced by all their CAD drawings,
and I'm trying to not have two copies of a 160GB dataset. That said, if
there's a strong performance gain to be had by retiling to a smaller size, I
can squeeze more space out of our RAID.

Steve

Stephen Mather, GIS Manager
Cleveland Metroparks
4101 Fulton Pkwy
Cleveland, OH 44144
svm@anonymised.com
Phone: (216) 635-3243
FAX: (216) 635-3286

-----Original Message-----
From: simboss1@anonymised.com [mailto:simboss1@anonymised.com] On Behalf Of Simone
Giannecchini
Sent: Wednesday, March 24, 2010 5:31 AM
To: svm@anonymised.com
Cc: geoserver-devel@lists.sourceforge.net
Subject: Re: [Geoserver-devel] Image pyramid improvements and tutorial

Ciao Stephen,
I downloaded the data and ran a quick test.
The pyramid creation was smooth, although I noticed that the geotiff files
you have provided us with do not contain a proper EPSG code for the CRS. I
picked one that was close enough and forced a reprojection (not the best
thing performance wise). Things worked for me in the end (although as I said
I have a few local changes that might have helped)

My questions for you are as follows:

- do you know a valid EPSG code for this dataset?
- why you using tiff that big for the single granule/tiles?

Ciao,
Simone.
-------------------------------------------------------
Ing. Simone Giannecchini
GeoSolutions S.A.S.
Founder - Software Engineer
Via Carignoni 51
55041 Camaiore (LU)
Italy

phone: +39 0584983027
fax: +39 0584983027
mob: +39 333 8128928

http://www.geo-solutions.it
http://geo-solutions.blogspot.com/
http://www.linkedin.com/in/simonegiannecchini
http://twitter.com/simogeo

-------------------------------------------------------

On Tue, Mar 23, 2010 at 3:00 PM, Stephen V. Mather
<svm@anonymised.com> wrote:

I should say, the file in question is called retile_test1.zip.

Steve

-----Original Message-----
From: Stephen V. Mather [mailto:svm@anonymised.com]
Sent: Monday, March 22, 2010 5:11 PM
To: 'Simone Giannecchini'
Cc: 'geoserver-devel@lists.sourceforge.net'
Subject: RE: [Geoserver-devel] Image pyramid improvements and tutorial

The test data are a little bigger than expected-- 195MB. They will be
available shortly at the following FTP site:

ftp://ftp.clevelandmetroparks.com/pdnr/out

Thanks,
Steve

Stephen Mather, GIS Manager
Cleveland Metroparks
4101 Fulton Pkwy
Cleveland, OH 44144
svm@anonymised.com
Phone: (216) 635-3243
FAX: (216) 635-3286

-----Original Message-----
From: simboss1@anonymised.com [mailto:simboss1@anonymised.com] On Behalf Of
Simone Giannecchini
Sent: Saturday, March 20, 2010 10:16 AM
To: svm@anonymised.com
Cc: geoserver-devel@lists.sourceforge.net
Subject: Re: [Geoserver-devel] Image pyramid improvements and tutorial

I guess that doing some testing with your data could be helpful. I
have some uncommitted fixes/improvements that might solve these
issues.

Simone

On 3/19/10, Stephen V. Mather <svm@anonymised.com> wrote:

Qcuik questions:
- are you adding a reprojection in the mix while you are doing this
tests?

I have tried it both with and without reprojection, to the same
effect.

- are all your granules of the same resolution?

Yes, they are all 0.15m pixels in 5000x4000 images. I have tried
both leaving the existing 0 level images at 5000x4000 (-pyramidOnly)
and retiling to 2048x2048, both with the same result.

Steve

On Tue, Mar 16, 2010 at 1:13 PM, Stephen Mather
<svm@anonymised.com> wrote:

Hi Andrea,
I've been working on giving it a spin with our own data. The
tutorial is excellent. Easy to read and follow.
I'm currently testing it on a Windows 2003 server with nightly
build geoserver-2.0.x-2010-03-03-bin.zip, and
geoserver-2.0.2-SNAPSHOT-pyramid-plugin.zip
The test I've been doing is with 0.5ft (0.15m) color
orthophotos that sum to about 150GB as TIFFs. They are currently
tiled as 5000x4000 images. I've retiled them with gdal_retile as
2048x2048 images with overviews with the following command:

gdal_retile -v -s_srs "EPSG:102722" -levels 5 -ps 2048 2048
-targetdir test3 --optfile tilelist.txt

I'm currently working on a subset of the data for testing. Checking
all the tiles generated from gdal_retile, they all look fine in an
image viewer, and check out (with a spot check) with gdalinfo.

When I load them in GeoServer, I get a strange effect. Some of the
tiles never render at their full resolution for a given zoom level,
but at a reduced resolution. If I load ~16 tiles vs. loading e.g.
40+ tiles the places where this happens shifts, i.e. tiles that were
a problem before stop being a problem, while other tiles exhibit the
same behavior.

Does this make sense, is this an effect you've seen, and would
images and log files be helpful?

Thanks,
Steve

Andrea Aime-4 wrote:

Hi all,
lately I've been working, under OpenGeo request, to improve the
situation vs image pyramids, in particular, making it easier to
configure a file based one.

...

Is anyone interested to give the tutorial a spin, a review? Is
there anything missing or that could be done better?

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

-------------------------------------------------------------------
-
-
---------
SOLARIS 10 is the OS for Data Centers - provides features such as
DTrace, Predictive Self Healing and Award Winning ZFS. Get Solaris
10 NOW http://p.sf.net/sfu/solaris-dev2dev
_______________________________________________
Geoserver-devel mailing list Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
View this message in context:

http://old.nabble.com/Image-pyramid-improvements-and-tutorial-tp27608
6
54p279
10740.html

Sent from the GeoServer - Dev mailing list archive at Nabble.com.

--------------------------------------------------------------------
-
-
--------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance. See
why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Geoserver-devel mailing list Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

---------------------------------------------------------------------
-
--------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance. See
why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Geoserver-devel mailing list Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

--
-------------------------------------------------------
Ing. Simone Giannecchini
GeoSolutions S.A.S.
Founder - Software Engineer
Via Carignoni 51
55041 Camaiore (LU)
Italy

phone: +39 0584983027
fax: +39 0584983027
mob: +39 333 8128928

http://www.geo-solutions.it http://geo-solutions.blogspot.com/
http://www.linkedin.com/in/simonegiannecchini
http://twitter.com/simogeo

-------------------------------------------------------

----------------------------------------------------------------------
--------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Andrea_Aime3 · March 29, 2010, 6:24pm

Stephen V. Mather ha scritto:

Hi Simone,
The correct CRS is EPSG:3728,
http://spatialreference.org/ref/epsg/3728/, although EPSG:102722 is pretty
close.
The size of the tiff is, well, either cheapness or laziness. The
current format, naming convention and size is what has been used by our
engineers in CAD for 8 years, and is referenced by all their CAD drawings,
and I'm trying to not have two copies of a 160GB dataset. That said, if
there's a strong performance gain to be had by retiling to a smaller size, I
can squeeze more space out of our RAID.

Hi Stephen,
tried another time, and I could also build the pyramid correctly, though
first I wiped out all the shapefiles and property files you did attach
so that the pyramd plugin could perform its job.

I can see the data just fine after that, though the rendering is quite
slow. The TIFFs are not really very well setup, they do have no inner
tiling, yet, I'm a bit surprised about the slowness.
Looked into it and the reason is that a reprojection is happening,
this is due to the native definition inside the GeoTiff not matching
the official EPSG one.

gdalinfo reports:

PROJCS["NAD_1983_StatePlane_Ohio_North_FIPS_3401_Feet",
     GEOGCS["NAD83",
         DATUM["North_American_Datum_1983",
             SPHEROID["GRS 1980",6378137,298.2572221010002,
                 AUTHORITY["EPSG","7019"]],
             AUTHORITY["EPSG","6269"]],
         PRIMEM["Greenwich",0],
         UNIT["degree",0.0174532925199433],
         AUTHORITY["EPSG","4269"]],
     PROJECTION["Lambert_Conformal_Conic_2SP"],
     PARAMETER["standard_parallel_1",40.43333333333333],
     PARAMETER["standard_parallel_2",41.7],
     PARAMETER["latitude_of_origin",39.66666666666666],
     PARAMETER["central_meridian",-82.5],
     PARAMETER["false_easting",1968500],
     PARAMETER["false_northing",0],
     UNIT["US survey foot",0.3048006096012192,
         AUTHORITY["EPSG","9003"]]]

whilst the EPSG database contains:

PROJCS["NAD83(NSRS2007) / Ohio North (ftUS)",
   GEOGCS["NAD83(NSRS2007)",
     DATUM["NAD83 (National Spatial Reference System 2007)",
       SPHEROID["GRS 1980", 6378137.0, 298.257222101, AUTHORITY["EPSG","7019"]],
       TOWGS84[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
       AUTHORITY["EPSG","6759"]],
     PRIMEM["Greenwich", 0.0, AUTHORITY["EPSG","8901"]],
     UNIT["degree", 0.017453292519943295],
     AXIS["Geodetic longitude", EAST],
     AXIS["Geodetic latitude", NORTH],
     AUTHORITY["EPSG","4759"]],
   PROJECTION["Lambert Conic Conformal (2SP)", AUTHORITY["EPSG","9802"]],
   PARAMETER["central_meridian", -82.5],
   PARAMETER["latitude_of_origin", 39.666666666666664],
   PARAMETER["standard_parallel_1", 41.7],
   PARAMETER["false_easting", 1968500.0],
   PARAMETER["false_northing", 0.0],
   PARAMETER["scale_factor", 1.0],
   PARAMETER["standard_parallel_2", 40.43333333333333],
   UNIT["foot_survey_us", 0.30480060960121924],
   AXIS["Easting", EAST],
   AXIS["Northing", NORTH],
   AUTHORITY["EPSG","3728"]]

As you can see there are quite a bit of parameters that are not equal.
To get decent performance you'd need to reproject the tiles to an officially supported EPSG code, or create a new one in $GEOSERVER_DATA_DIR/user_projections/epsg.properties that matches
1-1 the WKT of your data

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

Stephen_Mather2 · March 31, 2010, 9:25pm

Hmm, the embarassment of naïve configuration... . Let me do some
additional testing on my end. I assume then that you got the image pyramid
plugin to work without a special version of GeoServer?

Thanks,
Steve

-----Original Message-----
From: Andrea Aime [mailto:aaime@anonymised.com]
Sent: Monday, March 29, 2010 2:25 PM
To: svm@anonymised.com
Cc: 'Simone Giannecchini'; geoserver-devel@lists.sourceforge.net
Subject: Re: [Geoserver-devel] Image pyramid improvements and tutorial

Stephen V. Mather ha scritto:

Hi Simone,
The correct CRS is EPSG:3728,
http://spatialreference.org/ref/epsg/3728/, although EPSG:102722 is
pretty close.
The size of the tiff is, well, either cheapness or laziness. The
current format, naming convention and size is what has been used by
our engineers in CAD for 8 years, and is referenced by all their CAD
drawings, and I'm trying to not have two copies of a 160GB dataset.
That said, if there's a strong performance gain to be had by retiling
to a smaller size, I can squeeze more space out of our RAID.

Hi Stephen,
tried another time, and I could also build the pyramid correctly, though
first I wiped out all the shapefiles and property files you did attach so
that the pyramd plugin could perform its job.

I can see the data just fine after that, though the rendering is quite slow.
The TIFFs are not really very well setup, they do have no inner tiling, yet,
I'm a bit surprised about the slowness. Looked into it and the reason is
that a reprojection is happening, this is due to the native definition
inside the GeoTiff not matching the official EPSG one.

gdalinfo reports:

PROJCS["NAD_1983_StatePlane_Ohio_North_FIPS_3401_Feet",
     GEOGCS["NAD83",
         DATUM["North_American_Datum_1983",
             SPHEROID["GRS 1980",6378137,298.2572221010002,
                 AUTHORITY["EPSG","7019"]],
             AUTHORITY["EPSG","6269"]],
         PRIMEM["Greenwich",0],
         UNIT["degree",0.0174532925199433],
         AUTHORITY["EPSG","4269"]],
     PROJECTION["Lambert_Conformal_Conic_2SP"],
     PARAMETER["standard_parallel_1",40.43333333333333],
     PARAMETER["standard_parallel_2",41.7],
     PARAMETER["latitude_of_origin",39.66666666666666],
     PARAMETER["central_meridian",-82.5],
     PARAMETER["false_easting",1968500],
     PARAMETER["false_northing",0],
     UNIT["US survey foot",0.3048006096012192,
         AUTHORITY["EPSG","9003"]]]

whilst the EPSG database contains:

PROJCS["NAD83(NSRS2007) / Ohio North (ftUS)",
   GEOGCS["NAD83(NSRS2007)",
     DATUM["NAD83 (National Spatial Reference System 2007)",
       SPHEROID["GRS 1980", 6378137.0, 298.257222101,
AUTHORITY["EPSG","7019"]],
       TOWGS84[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
       AUTHORITY["EPSG","6759"]],
     PRIMEM["Greenwich", 0.0, AUTHORITY["EPSG","8901"]],
     UNIT["degree", 0.017453292519943295],
     AXIS["Geodetic longitude", EAST],
     AXIS["Geodetic latitude", NORTH],
     AUTHORITY["EPSG","4759"]],
   PROJECTION["Lambert Conic Conformal (2SP)", AUTHORITY["EPSG","9802"]],
   PARAMETER["central_meridian", -82.5],
   PARAMETER["latitude_of_origin", 39.666666666666664],
   PARAMETER["standard_parallel_1", 41.7],
   PARAMETER["false_easting", 1968500.0],
   PARAMETER["false_northing", 0.0],
   PARAMETER["scale_factor", 1.0],
   PARAMETER["standard_parallel_2", 40.43333333333333],
   UNIT["foot_survey_us", 0.30480060960121924],
   AXIS["Easting", EAST],
   AXIS["Northing", NORTH],
   AUTHORITY["EPSG","3728"]]

As you can see there are quite a bit of parameters that are not equal. To
get decent performance you'd need to reproject the tiles to an
officially supported EPSG code, or create a new one in
$GEOSERVER_DATA_DIR/user_projections/epsg.properties that matches 1-1 the
WKT of your data

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

Simone_Giannecchini3 · April 1, 2010, 6:12am

On Wed, Mar 31, 2010 at 11:25 PM, Stephen V. Mather
<svm@anonymised.com> wrote:

Hmm, the embarassment of naïve configuration... . Let me do some
additional testing on my end. I assume then that you got the image pyramid
plugin to work without a special version of GeoServer?

Yes, at least here.
I did not spend too much time on testing yet (not a lot of spare time
around here) but I just used a 2.0.x nightly.

Simone.

Thanks,
Steve

-----Original Message-----
From: Andrea Aime [mailto:aaime@anonymised.com]
Sent: Monday, March 29, 2010 2:25 PM
To: svm@anonymised.com
Cc: 'Simone Giannecchini'; geoserver-devel@lists.sourceforge.net
Subject: Re: [Geoserver-devel] Image pyramid improvements and tutorial

Stephen V. Mather ha scritto:

Hi Simone,
The correct CRS is EPSG:3728,
http://spatialreference.org/ref/epsg/3728/, although EPSG:102722 is
pretty close.
The size of the tiff is, well, either cheapness or laziness. The
current format, naming convention and size is what has been used by
our engineers in CAD for 8 years, and is referenced by all their CAD
drawings, and I'm trying to not have two copies of a 160GB dataset.
That said, if there's a strong performance gain to be had by retiling
to a smaller size, I can squeeze more space out of our RAID.

Hi Stephen,
tried another time, and I could also build the pyramid correctly, though
first I wiped out all the shapefiles and property files you did attach so
that the pyramd plugin could perform its job.

I can see the data just fine after that, though the rendering is quite slow.
The TIFFs are not really very well setup, they do have no inner tiling, yet,
I'm a bit surprised about the slowness. Looked into it and the reason is
that a reprojection is happening, this is due to the native definition
inside the GeoTiff not matching the official EPSG one.

gdalinfo reports:

PROJCS["NAD_1983_StatePlane_Ohio_North_FIPS_3401_Feet",
GEOGCS["NAD83",
DATUM["North_American_Datum_1983",
SPHEROID["GRS 1980",6378137,298.2572221010002,
AUTHORITY["EPSG","7019"]],
AUTHORITY["EPSG","6269"]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433],
AUTHORITY["EPSG","4269"]],
PROJECTION["Lambert_Conformal_Conic_2SP"],
PARAMETER["standard_parallel_1",40.43333333333333],
PARAMETER["standard_parallel_2",41.7],
PARAMETER["latitude_of_origin",39.66666666666666],
PARAMETER["central_meridian",-82.5],
PARAMETER["false_easting",1968500],
PARAMETER["false_northing",0],
UNIT["US survey foot",0.3048006096012192,
AUTHORITY["EPSG","9003"]]]

whilst the EPSG database contains:

PROJCS["NAD83(NSRS2007) / Ohio North (ftUS)",
GEOGCS["NAD83(NSRS2007)",
DATUM["NAD83 (National Spatial Reference System 2007)",
SPHEROID["GRS 1980", 6378137.0, 298.257222101,
AUTHORITY["EPSG","7019"]],
TOWGS84[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
AUTHORITY["EPSG","6759"]],
PRIMEM["Greenwich", 0.0, AUTHORITY["EPSG","8901"]],
UNIT["degree", 0.017453292519943295],
AXIS["Geodetic longitude", EAST],
AXIS["Geodetic latitude", NORTH],
AUTHORITY["EPSG","4759"]],
PROJECTION["Lambert Conic Conformal (2SP)", AUTHORITY["EPSG","9802"]],
PARAMETER["central_meridian", -82.5],
PARAMETER["latitude_of_origin", 39.666666666666664],
PARAMETER["standard_parallel_1", 41.7],
PARAMETER["false_easting", 1968500.0],
PARAMETER["false_northing", 0.0],
PARAMETER["scale_factor", 1.0],
PARAMETER["standard_parallel_2", 40.43333333333333],
UNIT["foot_survey_us", 0.30480060960121924],
AXIS["Easting", EAST],
AXIS["Northing", NORTH],
AUTHORITY["EPSG","3728"]]

As you can see there are quite a bit of parameters that are not equal. To
get decent performance you'd need to reproject the tiles to an
officially supported EPSG code, or create a new one in
$GEOSERVER_DATA_DIR/user_projections/epsg.properties that matches 1-1 the
WKT of your data

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Geoserver-devel mailing list
Geoserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-devel

Andrea_Aime3 · April 1, 2010, 6:31am

Stephen V. Mather ha scritto:

Hmm, the embarassment of naïve configuration... . Let me do some
additional testing on my end. I assume then that you got the image pyramid
plugin to work without a special version of GeoServer?

Correct, just took a nightly build, pointed it at the cleaned up
"test" directory (the first one that has tiles inside), after
removing all shapefiles, all property files.

However performance wise the result is not really usable, the
reprojection is quite heavy

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

Stephen_Mather2 · April 1, 2010, 1:12pm

Interesting that the nightly build I used really didn't work. I'll try a
more recent one and let you know how it goes.

I understand the performance issue. I'll add internal tiling to the tiffs
in place, do some tests with our other software that accesses it, and if
everything checks out, I won't have to duplicate the dataset. As this is a
static dataset, I wasn't too worried about performance-- once I have the
cache populated, I never have to populate it again, but that may be
short-sighted if I end up combining the aerials imagery with other overlays
that do change sometime down the road.

Thanks again,
Steve

Stephen Mather, GIS Manager
Cleveland Metroparks
4101 Fulton Pkwy
Cleveland, OH 44144
svm@anonymised.com
Phone: (216) 635-3243
FAX: (216) 635-3286

-----Original Message-----
From: Andrea Aime [mailto:aaime@anonymised.com]
Sent: Thursday, April 01, 2010 2:32 AM
To: svm@anonymised.com
Cc: 'Simone Giannecchini'; geoserver-devel@lists.sourceforge.net
Subject: Re: [Geoserver-devel] Image pyramid improvements and tutorial

Stephen V. Mather ha scritto:

Hmm, the embarassment of naïve configuration... . Let me do some
additional testing on my end. I assume then that you got the image
pyramid plugin to work without a special version of GeoServer?

Correct, just took a nightly build, pointed it at the cleaned up "test"
directory (the first one that has tiles inside), after removing all
shapefiles, all property files.

However performance wise the result is not really usable, the reprojection
is quite heavy

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.