Teh Fishy Blog

Overview

This is a problem I ran into while working on my team, and for some reason the solution was a bit involved. It's possible I didn't find the right resources at first, possibly because there seem to be different solutions depending on which version of OS X you are running.

My problems came in a couple forms, 1 running all of the unit tests in our Ruby of Rails project may have been a lot. Our project includes about 322 unit and functional test files with about 102 gems in the bundle/gemset.

It may have also happened when say, switching rvm gemsets, trying to open the Rails console, especially if it worked before, and then I added/updated gems at all, or running some other form of rails/rvm generic or test command. Any of those things may need to load a lot of files such as the gems, and helped me reach that open file handle limit.

I will attempt to capture my notes here so that if you run into the same type of problem, you can solve it as well.

Help Information

Brief Guide

# Source: https://superuser.com/questions/827984/open-files-limit-does-not-work-as-before-in-osx-yosemite

# Paired with: https://superuser.com/questions/302754/increase-the-maximum-number-of-open-file-descriptors-in-snow-leopard


# Doesn't seem to apply, at least what is viewed through 'sudo launchctl limit'

# Maybe this did successfully change the kernel limit, and I just need to change the [session] limit now?

# Also ran 'sudo launchctl limit maxfiles unlimited unlimited', which defaults to a maximum of 10240.

# But then I tried 'sudo launchctl limit maxfiles 1000000 1000000', and those values 'stuck'.


kern.sysv.shmmax=1073741824
kern.sysv.shmmin=1
kern.sysv.shmmni=4096
kern.sysv.shmseg=32
kern.sysv.shmall=1179648
kern.maxfilesperproc=65536
kern.maxfiles=65536

# Source: https://superuser.com/questions/302754/increase-the-maximum-number-of-open-file-descriptors-in-snow-leopard

# Paired with: https://superuser.com/questions/827984/open-files-limit-does-not-work-as-before-in-osx-yosemite


limit maxfiles 1000000 1000000

No ulimit command use necessary.

More Details

With this current set-up, I thought I had restarted my computer, re-run the problematic task (full unit/functional test suite), and still encountered the same file handle maximum error.

I just looked into the usual terminal environment configuration files (~/.config/fish/config.fish ~/.bashrc ~/.bash_profile ~/.profile /etc/profile /etc/bashrc), and I don't see any ulimit lines in there.

I thought that was required to have the settings take full effect, as I did it with those files set and ran the command in a terminal session to see the eventual success of the task, but I have been running the task after that point, having restarted my computer since then, without re-issuing the ulimit command again, and it works now, so maybe it's not needed?

Resource Dump

• SuperUser - Increase the maximum number of open file descriptors in Snow Leopard?

  • CLI (temporary)
  • sudo launchctl limit maxfiles 1000000 1000000
    • sudo ulimit -n 12288
    • or ulimit -n 12288 ?
  • Persist
    • /etc/launchd.conf
    • limit maxfiles 1000000 1000000
      • or limit maxfiles 8192 20480
    • limit maxproc 1000 2000
    • /etc/sysctl.conf
    • kern.maxfiles=20480
    • /etc/profile
    • ulimit -n 4096
  • Notes (implicitly quoted) [2]:
    • Alas: "The /etc/launchd.conf file is no longer consulted for subcommands to run during early boot time; this functionality was removed for security considerations."
    • You will need to restart for these changes to take effect.
    • AFAIK you can no longer set limits to 'unlimited' under OS X.
    • launchctl maxfiles is bounded by sysctl maxfiles, and therefore cannot exceed them.
    • sysctl seems to inherit kern.maxfilesperproc from launchctl maxfiles.
    • ulimit seems to inherit it's 'open files' value from launchctl by default.
    • You can set a custom ulimit within /etc/profile, or ~/.profile; while this isn't required, I've provided an example.
    • Be cautious when setting any of these values to a very high number when compared with their default - the features exist stability/security. I've taken these example numbers that I believe to be reasonable, written on other websites.

• Open files limit does not work as before in OSX Yosemite

  • CLI
    • sysctl -w kern.maxfiles=65000
  • CLI (alternate answer)

    echo 'kern.maxfiles=20480' | sudo tee -a /etc/sysctl.conf
    echo -e 'limit maxfiles 8192 20480\nlimit maxproc 1000 2000' | sudo tee -a /etc/launchd.conf
    echo 'ulimit -n 4096' | sudo tee -a /etc/profile
    

• Maximum number of open filehandles per process on OSX (and how to increase)

  • CLI
    • sysctl -w kern.maxfilesperproc=20000

• How to persist ulimit settings in OSX Mavericks?

  • Persist
    • /Library/LaunchDaemons/limit.maxfiles.plist
      • ....

Other

Right now I don't have any Windows instructions. I wonder if there is a similar problem in a pure-Windows environment, or the exact same problem when running in a *nix-like environment or interpreter such as Cygwin and Bash/Fish respectively.