[SunHELP] Apache/Solaris8 issues [(24)Too many open files: file permissions deny server access]

[Elwood Blues]

I tried the users at httpd.apache.org mailinglist, but no response,
maybe someone here has run into the same problem. Any help would
be greatly appreciated.

---------- Forwarded message ----------

A client of mine is having issues with apache running out of file
descriptors. After digging thru several list archives and newsgroups with
no luck, we decided to try this list. All the searches we did only
returned simular issues that dealt with jserv or apache-2.0, but we are
running a fairly plain setup:

Solaris 8 w/ the latest patch cluster
------
% uname -a
SunOS w3b.jukno.com 5.8 Generic_108528-14 sun4u sparc SUNW,UltraSPARC-IIi-Engine
% grep _fd /etc/system
set rlim_fd_max = 166384
set rlim_fd_cur = 8192
% ulimit -a | grep files
open files                  8192

Apache 1.3.14
------
% /apache/sbin/httpd -l
Compiled-in modules:
  http_core.c
  mod_env.c
  mod_log_config.c
  mod_mime.c
  mod_negotiation.c
  mod_include.c
  mod_autoindex.c
  mod_dir.c
  mod_cgi.c
  mod_asis.c
  mod_imap.c
  mod_actions.c
  mod_userdir.c
  mod_alias.c
  mod_access.c
  mod_auth.c
  mod_auth_db.c
  mod_so.c
  mod_setenvif.c
  mod_auth_mysql.c
suexec: enabled; valid wrapper /apache/sbin/suexec
% /apache/sbin/httpd -v
Server version: Apache/1.3.14 (Unix)
Server built:   Jan 17 2001 03:06:02

The only thing non-standard that we have on this server is that we
are piping logging thru a logfile rotation program called httplog
(http://nutbar.chemlab.org/downloads/programs/httplog-2.1.tar.gz)
like so (in VirtualHost directives):

CustomLog "|/usr/local/sbin/httplog -z \
    -s /apache/sites//logs/access_log \
    /apache/sites/a.b.c.d/logs/access_log.%Y%m" combined
ErrorLog "|/usr/local/sbin/httplog -z \
    -s /apache/sites/a.b.c.d/logs/error_log \
    /apache/sites/a.b.c.d/logs/error_log.%Y%m"
CustomLog "|/usr/local/sbin/httplog -z \
    -s /apache/sites/a.b.c.d/logs/bandwidth_log \
    /apache/sites/a.b.c.d/logs/bandwidth_log.%Y%m" bandwidth

This creates a logfile that is named by the current date (ex.
access_log.200205) with a symlink pointing to the current file.

At around 45 or fewer VirtualHosts, we have no issues. Once we get above
that amount, browsers get a 403 error code and we start getting the
following errors in the VirtualHosts error_logs:
[Tue Jun 11 19:37:00 2002] [error] [client w.x.y.z] (24)Too many open files: file permissions deny server access: /apache/sites/a.b.c.d/htdocs/index.html

If we turn off one of the logging statements, say leaving just access and
error log, in all the VirtualHosts and restart apache, pages are served
properly and no more errors. Initially, /etc/system was configured as:
set rlim_fd_max = 1024
set rlim_fd_cur = 256
so we upped it as noted above. We also confirmed that the new settings
are in effect by running testfd [1]:
% testfd
opened file #8176
You can open 8189 files simultaneously.

For whatever reason, the client does not want to upgrade to 1.3.24. I know
there must be something I am not taking into consideration, has anyone
else run into an issue like this before?

Thanks in advance,

    -F

Notes:
[1] testfd.c code
% more testfd.c
#include <stdio.h>
#include <unistd.h>
#include <memory.h>
#include <string.h>
#include <errno.h>
#include <sys/fcntl.h>

#define MAX 65535 /* mind, (short) -1 == (ushort) 65535 */

static int fds[MAX];

int
main( int argc, char* argv[] )
{
  int i, fd;

  memset( fds, 0, sizeof(fds) );
  setbuf( stdout, 0 );

  for ( i=0; i<MAX; i++ ) {
    if ( (fds[i]=open( argv[0], O_RDONLY )) == -1 ) {
      if ( errno != EMFILE ) perror( "cannot open" );
      break;
    } else {
      if ( (i & 15) == 0 ) printf( "\ropened file #%d", i );
    }
  }

  printf( "\nYou can open %d files simultaneously.\n", i );
  for ( i--; i>=0; i-- ) close(fds[i]);
  return 0;
}