[GRASS-dev] G_open_cell_old: unable to open raster map

GRASS GRASS developer
1

Greetings-

I’ve been forced in to development by this bug (ie i have a only a vague idea of what I’m doing). Help greatly appreciated (need to work on thesis not bugs).

Summary:

I’ve generated a location and a mapset, in this mapset I have files named
input_file.0001, … , input_file.nnnn, … , input_file.3313

I’m running r.hydro.CASC2 (used to be in GRASS), which i’ve built from gipe add-on package.
When it gets to approximately nnnn=1016, i get (i’ll use the real rast and mapset names, in case they are relevant):

WARNING: Unable to open raster map <SBB_runoff_lumped_05.1016@lumped_05_9930>

The offended block of code in r.hydro.CASC2D is

if((radar_fd[nfile]=G_open_cell_old(radar_file[nfile],mapset))<0)
{
sprintf(buf,“cannot open [%s]\n”, radar_file[nfile]);
G_fatal_error(buf);
exit(1);
}

But the file is right in place, looking all fine and dandy. I’ve also tried copying input_file.1015 to input_file.1016 using both the gui and on command line. I still get no error for 1015, only for 1016. Also, if i comment

I’m at a complete loss. Is there a bug in G_open_cell_old? Something which suggests otherwise: if i comment out the above code, CASC2D apparently runs until that same map is to be read by G_get_map_row(), at which point I dont even get a warning, but a seg fault. )

Thanks in advance.

James

–

James McCreight mccreigh @at@ colorado.edu
NASA Earth Science Fellow
PhD Candidate, Water Resources and Environmental Fluid Mechanics
University of Colorado, Boulder
cell: (831) 261-5149

VoIP (to cell): (720) 897-7546

2

James McCreight wrote:

I've been forced in to development by this bug (ie i have a only a vague
idea of what I'm doing). Help greatly appreciated (need to work on thesis
not bugs).

Summary:

I've generated a location and a mapset, in this mapset I have files named
input_file.0001, ... , input_file.nnnn, ... , input_file.3313

I'm running r.hydro.CASC2 (used to be in GRASS), which i've built from gipe
add-on package.
When it gets to approximately nnnn=1016, i get (i'll use the real rast and
mapset names, in case they are relevant):

WARNING: Unable to open raster map
<SBB_runoff_lumped_05.1016@lumped_05_9930>

If it's happening after ~1000 files, I strongly suspect you're hitting
an OS limit on the number of open file descriptors. It's common to
have a limit of 1024 open files per process.

Run "ulimit -n" in the shell to determine the soft limit. If this is
too low, use "ulimit -Hn" to determine the hard limit. The soft limit
is what the OS actually checks against; you can change the soft limit
with e.g. "ulimit -n 5000", but you cannot increase it above the hard
limit, and a non-root user cannot increase the hard limit.

On systems which use PAM, these limits are set in the file
/etc/security/limits.conf and any files in the directory
/etc/security/limits.d. This file is read by PAM (specifically, the
pam_limits.so module) at login, which sets the limits before dropping
root privilege and switching to the logged in user.

If you have the ability to change that file, a line such as:

  * hard nofile 5000

will increase the hard limit to 5000. The limit will take effect for
subsequent logins (you can run "login" in an xterm to get a shell with
the new limits if you don't want to quit your current desktop
session).

--
Glynn Clements <glynn@gclements.plus.com>

3

Thanks Glynn-

that’s exactly the problem. I’ve never heard of ulimit (I try not to keep so many files open!). SO glad I know!

Cheers,

James

On Wed, Aug 18, 2010 at 8:11 PM, Glynn Clements <glynn@gclements.plus.com> wrote:

James McCreight wrote:

I’ve been forced in to development by this bug (ie i have a only a vague
idea of what I’m doing). Help greatly appreciated (need to work on thesis
not bugs).

Summary:

I’ve generated a location and a mapset, in this mapset I have files named
input_file.0001, … , input_file.nnnn, … , input_file.3313

I’m running r.hydro.CASC2 (used to be in GRASS), which i’ve built from gipe
add-on package.
When it gets to approximately nnnn=1016, i get (i’ll use the real rast and
mapset names, in case they are relevant):

WARNING: Unable to open raster map
<SBB_runoff_lumped_05.1016@lumped_05_9930>

If it’s happening after ~1000 files, I strongly suspect you’re hitting
an OS limit on the number of open file descriptors. It’s common to
have a limit of 1024 open files per process.

Run “ulimit -n” in the shell to determine the soft limit. If this is
too low, use “ulimit -Hn” to determine the hard limit. The soft limit
is what the OS actually checks against; you can change the soft limit
with e.g. “ulimit -n 5000”, but you cannot increase it above the hard
limit, and a non-root user cannot increase the hard limit.

On systems which use PAM, these limits are set in the file
/etc/security/limits.conf and any files in the directory
/etc/security/limits.d. This file is read by PAM (specifically, the
pam_limits.so module) at login, which sets the limits before dropping
root privilege and switching to the logged in user.

If you have the ability to change that file, a line such as:

  • hard nofile 5000

will increase the hard limit to 5000. The limit will take effect for
subsequent logins (you can run “login” in an xterm to get a shell with
the new limits if you don’t want to quit your current desktop
session).

–
Glynn Clements <glynn@gclements.plus.com>

–

James McCreight mccreigh @at@ colorado.edu
NASA Earth Science Fellow
PhD Candidate, Water Resources and Environmental Fluid Mechanics
University of Colorado, Boulder
cell: (831) 261-5149

VoIP (to cell): (720) 897-7546

4

On Thu, Aug 19, 2010 at 4:54 AM, James McCreight <mccreigh@gmail.com> wrote:

Thanks Glynn-
that's exactly the problem. I've never heard of ulimit (I try not to keep so
many files open!). SO glad I know!

We have some documentation here:
http://grass.osgeo.org/wiki/Memory_issues

Please expand it, if needed.

thanks
Markus