[Geoserver-users] ImageMosaicReader and FileNotFoundException (Too many open files)

2StepForward · October 13, 2008, 12:32pm

Hello,

I have read this thread (ImageMosaicReader and FileNotFoundException (Too
many open files))in the geotools forum and I also get this error with
GeoServer 1.6.3 and a using of ImageMosaic.
I have a shape-file which references up to 3000 files. If I take a client
request over a scale which includes more than 1024 files, than GeoServer
crashes with the exception "too many open files".
The path table of the process in /proc/ID/fd shows all the 1024 open pathes
to the image files.

I have read in another thread, that Simone had wrote:

Ciao Stefan,
your analysis is correct, this is indeed an issue that we have to
solve within geotools.
Would you mind opening up a JIRA bug for this problem so that next
week I can work on it?

but I could't find any bug report which discribe the bug fix.
Has anyone a solution for this problem?

Cheers,
Klaus
--
View this message in context: http://www.nabble.com/ImageMosaicReader-and-FileNotFoundException-(Too-many-open-files)-tp19954017p19954017.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

Andrea_Aime3 · October 13, 2008, 12:54pm

2StepForward ha scritto:

Hello,

I have read this thread (ImageMosaicReader and FileNotFoundException (Too
many open files))in the geotools forum and I also get this error with
GeoServer 1.6.3 and a using of ImageMosaic.
I have a shape-file which references up to 3000 files. If I take a client
request over a scale which includes more than 1024 files, than GeoServer
crashes with the exception "too many open files".
The path table of the process in /proc/ID/fd shows all the 1024 open pathes
to the image files.

I have in another thread, thar Simone had wrote:

Ciao Stefan,
your analysis is correct, this is indeed an issue that we have to
solve within geotools.
Would you mind opening up a JIRA bug for this problem so that next
week I can work on it?

but I could't find any bug report which discribe the bug fix.

Did a quick search, nobody opened a jira issue. Which is the
proper recipe to make sure the bug won't be fixed, developers
cannot remember every single issue reported on the ml

Has anyone a solution for this problem?

Unless the reader caching strategy has changed (Simone?), I guess
the only way to help is to use ulimit to bump up the
number of files a process can open at the same time.

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

2StepForward · October 13, 2008, 1:27pm

Hi Andrea,

and thanks for reply. Is there a possibility on the Server side, meens in
GeoServer, to protect the Server from these overdone requests. Like a min-
and max scale definition on a coverage layer.
Problem is, i have to protect the server from these kinds of massive
requests and the olny way I see is to limit the possible scales for the
coverage layer.

I work with OL and there is a possibility the set up min- and max scales,
but if I use this feature then the client will protect the server from
possible damaging requests.

Cheers Klaus

Andrea Aime-4 wrote:

2StepForward ha scritto:

Hello,

I have read this thread (ImageMosaicReader and FileNotFoundException (Too
many open files))in the geotools forum and I also get this error with
GeoServer 1.6.3 and a using of ImageMosaic.
I have a shape-file which references up to 3000 files. If I take a client
request over a scale which includes more than 1024 files, than GeoServer
crashes with the exception "too many open files".
The path table of the process in /proc/ID/fd shows all the 1024 open
pathes
to the image files.

I have in another thread, thar Simone had wrote:

Ciao Stefan,
your analysis is correct, this is indeed an issue that we have to
solve within geotools.
Would you mind opening up a JIRA bug for this problem so that next
week I can work on it?

but I could't find any bug report which discribe the bug fix.

Did a quick search, nobody opened a jira issue. Which is the
proper recipe to make sure the bug won't be fixed, developers
cannot remember every single issue reported on the ml

Has anyone a solution for this problem?

Unless the reader caching strategy has changed (Simone?), I guess
the only way to help is to use ulimit to bump up the
number of files a process can open at the same time.

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
View this message in context: http://www.nabble.com/ImageMosaicReader-and-FileNotFoundException-(Too-many-open-files)-tp19954017p19954907.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

Andrea_Aime3 · October 13, 2008, 1:37pm

2StepForward ha scritto:

Hi Andrea,

and thanks for reply. Is there a possibility on the Server side, meens in
GeoServer, to prodect the Server from thees overdone requests. Like a min-
and max scale definition on a coverage layer.
Problem is, i have to prodect the server from thees kinds of massive
requests and the olny way I see is to limit the possible scales for the
coverage layer.

I work with OL and there is a possibility the set up min- and max scales,
but if I use this feature then the client will protect the server from
possible damaging requests.

I can give you an improvement, but not a general solution.
You can build an SLD for your raster data with min and max scale
denominator rules that will make the layer appear only within a certain
scale range. You can find an example in the style of New York roads
layers that you can find in the main GeoServer distribution.
Yet, this won't protect you against requests in which the user:
- manually chooses another raster style that does not have such rules
   (this can be prevented by removing the other raster styles if
    possible)
- provides a custom SLD along the request using &sld=... or
   &sld_body=...
   and in this case there is really nothing that can be done.

I've opened a jira issue quite some time ago with the precise
intent of looking into this kind of quality of service issues
http://jira.codehaus.org/browse/GEOS-1127
but the fixes are not trivial and no one stepped up to actually
code it or to sponsor such developments (on the contrary, I've been
packed with other kind of sponsored requests for most of
the past year).

Cheers
Andrea

--
Andrea Aime
OpenGeo - http://opengeo.org
Expert service straight from the developers.

Simone3 · October 13, 2008, 1:41pm

Ciao Klaus,
please read below...

On Mon, Oct 13, 2008 at 2:32 PM, 2StepForward <kwegezeder@anonymised.com> wrote:

Hello,

I have read this thread (ImageMosaicReader and FileNotFoundException (Too
many open files))in the geotools forum and I also get this error with
GeoServer 1.6.3 and a using of ImageMosaic.
I have a shape-file which references up to 3000 files. If I take a client
request over a scale which includes more than 1024 files, than GeoServer
crashes with the exception "too many open files".
The path table of the process in /proc/ID/fd shows all the 1024 open pathes
to the image files.

I have read in another thread, that Simone had wrote:

Ciao Stefan,
your analysis is correct, this is indeed an issue that we have to
solve within geotools.
Would you mind opening up a JIRA bug for this problem so that next
week I can work on it?

but I could't find any bug report which discribe the bug fix.
Has anyone a solution for this problem?

You could not find it because you did not open it :-).
The error you are getting is related to the fact that actually the
ImageMosaic plugin right now works only in load-deferred mode, which
means it opens up all the files it need and then it start streaming
one of theam at the time. The quick fix could be allowing the reader
to work in immediate mode.
However, I think you should consider either using a pyramid or using
the MinScale MaxScale param of the SLD to make this layer
appear/disappear at correct scale, since, regardless of whether or not
we change the execution model for the MosaicReader, you would be
opening too many files at the same time and performances will suffer.
Anyway, please, do open a BUG report for this issue.

Cheers,
Simone.

Cheers,
Klaus
--
View this message in context: http://www.nabble.com/ImageMosaicReader-and-FileNotFoundException-(Too-many-open-files)-tp19954017p19954017.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
-------------------------------------------------------
Eng. Simone Giannecchini
GeoSolutions S.A.S.
Owner - Software Engineer
Via Carignoni 51
55041 Camaiore (LU)
Italy

phone: +39 0584983027
fax: +39 0584983027
mob: +39 333 8128928

http://www.geo-solutions.it
http://www.geo-solutions.it/simone.giannecchini
http://www.linkedin.com/in/simonegiannecchini

-------------------------------------------------------

2StepForward · October 13, 2008, 4:04pm

Hi Simone,

I will try it to protect with the SLD, this works for me great.
Thank you and Andrea too for the tip.

Cheers Klaus

Simone.Giannecchini wrote:

Ciao Klaus,
please read below...

On Mon, Oct 13, 2008 at 2:32 PM, 2StepForward <kwegezeder@anonymised.com> wrote:

Hello,

I have read this thread (ImageMosaicReader and FileNotFoundException (Too
many open files))in the geotools forum and I also get this error with
GeoServer 1.6.3 and a using of ImageMosaic.
I have a shape-file which references up to 3000 files. If I take a client
request over a scale which includes more than 1024 files, than GeoServer
crashes with the exception "too many open files".
The path table of the process in /proc/ID/fd shows all the 1024 open
pathes
to the image files.

I have read in another thread, that Simone had wrote:

Ciao Stefan,
your analysis is correct, this is indeed an issue that we have to
solve within geotools.
Would you mind opening up a JIRA bug for this problem so that next
week I can work on it?

but I could't find any bug report which discribe the bug fix.
Has anyone a solution for this problem?

You could not find it because you did not open it :-).
The error you are getting is related to the fact that actually the
ImageMosaic plugin right now works only in load-deferred mode, which
means it opens up all the files it need and then it start streaming
one of theam at the time. The quick fix could be allowing the reader
to work in immediate mode.
However, I think you should consider either using a pyramid or using
the MinScale MaxScale param of the SLD to make this layer
appear/disappear at correct scale, since, regardless of whether or not
we change the execution model for the MosaicReader, you would be
opening too many files at the same time and performances will suffer.
Anyway, please, do open a BUG report for this issue.

Cheers,
Simone.

Cheers,
Klaus
--
View this message in context:
http://www.nabble.com/ImageMosaicReader-and-FileNotFoundException-(Too-many-open-files)-tp19954017p19954017.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
-------------------------------------------------------
Eng. Simone Giannecchini
GeoSolutions S.A.S.
Owner - Software Engineer
Via Carignoni 51
55041 Camaiore (LU)
Italy

phone: +39 0584983027
fax: +39 0584983027
mob: +39 333 8128928

http://www.geo-solutions.it
http://www.geo-solutions.it/simone.giannecchini
http://www.linkedin.com/in/simonegiannecchini

-------------------------------------------------------

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
View this message in context: http://www.nabble.com/ImageMosaicReader-and-FileNotFoundException-(Too-many-open-files)-tp19954017p19957765.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

Simone3 · October 13, 2008, 4:37pm

Ciao Klaus,
please, file the bug report so that I can keep track of this issue/RFE.

Simone.

On Mon, Oct 13, 2008 at 6:04 PM, 2StepForward <kwegezeder@anonymised.com> wrote:

Hi Simone,

I will try it to protect with the SLD, this works for me great.
Thank you and Andrea too for the tip.

Cheers Klaus

Simone.Giannecchini wrote:

Ciao Klaus,
please read below...

On Mon, Oct 13, 2008 at 2:32 PM, 2StepForward <kwegezeder@anonymised.com> wrote:

Hello,

I have read this thread (ImageMosaicReader and FileNotFoundException (Too
many open files))in the geotools forum and I also get this error with
GeoServer 1.6.3 and a using of ImageMosaic.
I have a shape-file which references up to 3000 files. If I take a client
request over a scale which includes more than 1024 files, than GeoServer
crashes with the exception "too many open files".
The path table of the process in /proc/ID/fd shows all the 1024 open
pathes
to the image files.

I have read in another thread, that Simone had wrote:

Ciao Stefan,
your analysis is correct, this is indeed an issue that we have to
solve within geotools.
Would you mind opening up a JIRA bug for this problem so that next
week I can work on it?

but I could't find any bug report which discribe the bug fix.
Has anyone a solution for this problem?

You could not find it because you did not open it :-).
The error you are getting is related to the fact that actually the
ImageMosaic plugin right now works only in load-deferred mode, which
means it opens up all the files it need and then it start streaming
one of theam at the time. The quick fix could be allowing the reader
to work in immediate mode.
However, I think you should consider either using a pyramid or using
the MinScale MaxScale param of the SLD to make this layer
appear/disappear at correct scale, since, regardless of whether or not
we change the execution model for the MosaicReader, you would be
opening too many files at the same time and performances will suffer.
Anyway, please, do open a BUG report for this issue.

Cheers,
Simone.

Cheers,
Klaus
--
View this message in context:
http://www.nabble.com/ImageMosaicReader-and-FileNotFoundException-(Too-many-open-files)-tp19954017p19954017.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
-------------------------------------------------------
Eng. Simone Giannecchini
GeoSolutions S.A.S.
Owner - Software Engineer
Via Carignoni 51
55041 Camaiore (LU)
Italy

phone: +39 0584983027
fax: +39 0584983027
mob: +39 333 8128928

http://www.geo-solutions.it
http://www.geo-solutions.it/simone.giannecchini
http://www.linkedin.com/in/simonegiannecchini

-------------------------------------------------------

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's
challenge
Build the coolest Linux based applications with Moblin SDK & win great
prizes
Grand prize is a trip for two to an Open Source event anywhere in the
world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
View this message in context: http://www.nabble.com/ImageMosaicReader-and-FileNotFoundException-(Too-many-open-files)-tp19954017p19957765.html
Sent from the GeoServer - User mailing list archive at Nabble.com.

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Geoserver-users mailing list
Geoserver-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geoserver-users

--
-------------------------------------------------------
Eng. Simone Giannecchini
GeoSolutions S.A.S.
Owner - Software Engineer
Via Carignoni 51
55041 Camaiore (LU)
Italy

phone: +39 0584983027
fax: +39 0584983027
mob: +39 333 8128928

http://www.geo-solutions.it
http://www.geo-solutions.it/simone.giannecchini
http://www.linkedin.com/in/simonegiannecchini

-------------------------------------------------------