[GRASS-dev] rast_open_new() saturates after opening 127 temporary files

GRASS GIS GRASS GIS developer
1

Hi,

Opening 127 temporary raster files is the limit of GRASS,

How can I get above that limit?

Cheers,
Yann

–

2

Yann Chemin wrote:

Opening 127 temporary raster files is the limit of GRASS,

How can I get above that limit?

You need to provide more information: version, OS, any error or debug
messages generated.

I don't have this problem with 7.0 on Linux.

But first, check that it isn't an OS issue. "ulimit -n" reports the
maximum number of open files (in 7.0, you need two per map). Also,
each completed raster needs a subdirectory within the cell_misc
directory, and some filesystems impose a limit on the number of
subdirectories (often much lower than the limit on the number of
files).

--
Glynn Clements <glynn@gclements.plus.com>

3

Hi,

Linux: Ubuntu 13.04
ulimit: open files (-n) 1024

from : http://www.itworld.com/operating-systems/317369/setting-limits-ulimit

-n The maximum number of open file descriptors (most systems
       do not allow this value to be set)

indeed:
ulimit -n unlimited
bash: ulimit: open files: cannot modify limit: Operation not permitted

Yann

On 29 May 2013 01:34, Glynn Clements <glynn@gclements.plus.com> wrote:

Yann Chemin wrote:

Opening 127 temporary raster files is the limit of GRASS,

How can I get above that limit?

You need to provide more information: version, OS, any error or debug
messages generated.

I don't have this problem with 7.0 on Linux.

But first, check that it isn't an OS issue. "ulimit -n" reports the
maximum number of open files (in 7.0, you need two per map). Also,
each completed raster needs a subdirectory within the cell_misc
directory, and some filesystems impose a limit on the number of
subdirectories (often much lower than the limit on the number of
files).

--
Glynn Clements <glynn@gclements.plus.com>

--
----

4

ah yes i am using G7 trunk

On 29 May 2013 08:04, Yann Chemin <ychemin@gmail.com> wrote:

Hi,

Linux: Ubuntu 13.04
ulimit: open files (-n) 1024

from : http://www.itworld.com/operating-systems/317369/setting-limits-ulimit

-n The maximum number of open file descriptors (most systems
       do not allow this value to be set)

indeed:
ulimit -n unlimited
bash: ulimit: open files: cannot modify limit: Operation not permitted

Yann

On 29 May 2013 01:34, Glynn Clements <glynn@gclements.plus.com> wrote:

Yann Chemin wrote:

Opening 127 temporary raster files is the limit of GRASS,

How can I get above that limit?

You need to provide more information: version, OS, any error or debug
messages generated.

I don't have this problem with 7.0 on Linux.

But first, check that it isn't an OS issue. "ulimit -n" reports the
maximum number of open files (in 7.0, you need two per map). Also,
each completed raster needs a subdirectory within the cell_misc
directory, and some filesystems impose a limit on the number of
subdirectories (often much lower than the limit on the number of
files).

--
Glynn Clements <glynn@gclements.plus.com>

--
----

--
----

5

Yann Chemin wrote:

Linux: Ubuntu 13.04
ulimit: open files (-n) 1024

In which case, that isn't the problem.

What type of filesystem are you using? ext2 has a limit of 32768
subdirectories, increased to 64000 in ext4. I know that btrfs has a
rather low limit on the number of hard links (which normally
determines the maximum number of subdirectories, as each
subdirectory's ".." entry is a hard link to its parent).

indeed:
ulimit -n unlimited
bash: ulimit: open files: cannot modify limit: Operation not permitted

You can't increase a soft limit above the corresponding hard limit,
and you can't increase a hard limit unless you're root (and sudo won't
work because the limits are per-process, so ulimit has to be a shell
built-in). On a system which uses PAM, the limits are normally set on
login based upon the contents of /etc/security/limits.conf.

--
Glynn Clements <glynn@gclements.plus.com>

6

On Wed, May 29, 2013 at 4:34 AM, Yann Chemin <ychemin@gmail.com> wrote:

Hi,

Linux: Ubuntu 13.04
ulimit: open files (-n) 1024

from : http://www.itworld.com/operating-systems/317369/setting-limits-ulimit

-n The maximum number of open file descriptors (most systems
       do not allow this value to be set)

indeed:
ulimit -n unlimited
bash: ulimit: open files: cannot modify limit: Operation not permitted

See also here:

http://grasswiki.osgeo.org/wiki/Large_raster_data_processing#Number_of_open_files_limitation

Markus

7

See here ( entry and comments)
http://www.cyberciti.biz/faq/linux-unix-nginx-too-many-open-files/

It’s a 3 step process on Ubuntu.

  1. Check the settings for fs.file-max in /etc/sysctl.conf
    2)Edit /etc/security/limits.conf and bump up the file limit for the userid that you run GRASS under. I bumped mine up to 10000 for hard and soft
  2. Add session required pam_limits.so to /etc/pam.d/common-session

Doug

···

On Wed, May 29, 2013 at 8:03 AM, Markus Neteler <neteler@osgeo.org> wrote:

On Wed, May 29, 2013 at 4:34 AM, Yann Chemin <ychemin@gmail.com> wrote:

Hi,

Linux: Ubuntu 13.04
ulimit: open files (-n) 1024

from : http://www.itworld.com/operating-systems/317369/setting-limits-ulimit

-n The maximum number of open file descriptors (most systems
do not allow this value to be set)

indeed:
ulimit -n unlimited
bash: ulimit: open files: cannot modify limit: Operation not permitted

See also here:

http://grasswiki.osgeo.org/wiki/Large_raster_data_processing#Number_of_open_files_limitation

Markus

grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

–

Doug Newcomb
USFWS
Raleigh, NC
919-856-4520 ext. 14 doug_newcomb@fws.gov

The opinions I express are my own and are not representative of the official policy of the U.S.Fish and Wildlife Service or Dept. of the Interior. Life is too short for undocumented, proprietary data formats.