Open file descriptor limits: is there a usable solution?

[michael-p-lang]

I've done a lot of searching, and I can't find a real solution to the old ulimit problem. The leading contender for a solution is #1576 (comment) except I have to manually do exec zsh every time. It would be nice if there was a permanent or otherwise hands-off solution to this problem.

I'm running Ubuntu-20.04 in WSL2
Windows Version 20H2 Build 19042.685

uname -a
Linux MY-COMPUTER 4.19.128-microsoft-standard #1 SMP Tue Jun 23 12:58:10 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[Delicious-Bacon]

Bump!

Increasing ulimit STILL requires manual intervention. I work with a lot of Windows Terminal tabs at the same time, and have to manually increase the limits in them whenever I open new tab. This is ridiculous.

[NotTheDr01ds]

@Delicious-Bacon and (if you're still around) @michael-p-lang

It's possible there's some confusion on those issues that were linked -- They were both WSL1 specific issues. The situation on WSL2 is much, much better.

Short answer: In WSL2 you should only need to add ulimit -Sn <new_limit> to your startup config (e.g. ~/.bashrc, ~/.zshrc, etc.). This will change the soft limit (up to 1,048,576) for both the current shell and all subprocesses launched in that shell.

More Details:

Look, there seems to be a lot of confusion out there in general around the open-files limit on Linux, so there's no surprise that the situation on WSL is even less clear.

There are (at least) three limits in place regarding number of open files. To start with, on Linux:

• The kernel-defined, system-wide limit for all processes (cat /proc/sys/fs/nr_open to see the value) defaults to 1,048,576.
• The per-process hard limit (uname -Hn) is initially set to 1,048,576.
  • A non-root user can lower this value, but not raise, this limit.
  • The root user may raise the limit, but not above the system-wide, kernel-defined limit above.
• The soft limit (uname -Sn) is initially set to 1,024.
  • Any user, root or non-root, can raise or lower this limit as long as it does not exceed the hard limit above.

There's probably a Windows limit in place as well to which WSL2 is subject, and perhaps one defined by Hyper-V. I can't speak to those numbers.

On Linux, and in WSL2, you will typically be constrained by the soft limit. My guess is that this is what you are trying to change. As mentioned above, as a end-user, you can do so with a simple ulimit -Sn <new_limit> as long as <new_limit> is less than the hard limit. For example:

$ ulimit -Sn
1024
$ ulimit -Hn
1048576
$ ulimit -Sn 2048
$ ulimit -Sn
2048
$ ulimit -Hn 4096
$ ulimit -Hn
4096
$ ulimit -Hn 8192
-bash: ulimit: open files: cannot modify limit: Operation not permitted
$ ulimit -Sn 8192
-bash: ulimit: open files: cannot modify limit: Invalid argument

Simply placing the ulimit -Sn <new_limit> in your startup config is sufficient for changing the limit for your interactive shell and all subprocesses started from it.

Proper ulimit usage

Best practice is to always use the -S or -H to specify hard or soft. Please note that ulimit -n <new_limit> (without specifying hard or soft) changes both the hard and soft at the same time, and that can be confusing. Again, once the hard limit is lowered by a non-root user, it cannot be raised again, so:

$ ulimit -n 1024
$ ulimit -n
1024
$ ulimit -n 2048
-bash: ulimit: open files: cannot modify limit: Operation not permitted

Testing the limits

Besides confirming the new limit with ulimit -Sn, you can test it quite easily using file descriptors. The system will not allow an fd which exceeds (or meets, since fd's are 0-based) the open-files-limit. For example:

$ ulimit -Sn
1024
$ exec 1023</dev/null
$ ls -l /proc/self/fd

total 0
lrwx------ 1 ntd ntd 64 Jul 29 10:53 0 -> /dev/pts/0
l-wx------ 1 ntd ntd 64 Jul 29 10:53 1 -> pipe:[1106]
lr-x------ 1 ntd ntd 64 Jul 29 10:53 1023 -> /dev/null
lrwx------ 1 ntd ntd 64 Jul 29 10:53 2 -> /dev/pts/0
lr-x------ 1 ntd ntd 64 Jul 29 10:53 3 -> /proc/171/fd
lrwx------ 1 ntd ntd 64 Jul 29 10:53 6 -> /dev/pts/0

$ exec 1024</dev/null
-bash: 1024: Bad file descriptor

$ ulimit -Sn 2048
$ exec 1024</dev/null
$ ls -l /proc/self/fd

total 0
lrwx------ 1 ntd ntd 64 Jul 29 10:54 0 -> /dev/pts/0
l-wx------ 1 ntd ntd 64 Jul 29 10:54 1 -> /dev/pts/0
lr-x------ 1 ntd ntd 64 Jul 29 10:54 1023 -> /dev/null
lr-x------ 1 ntd ntd 64 Jul 29 10:54 1024 -> /dev/null
lrwx------ 1 ntd ntd 64 Jul 29 10:54 2 -> /dev/pts/0
lr-x------ 1 ntd ntd 64 Jul 29 10:54 3 -> /proc/175/fd
lrwx------ 1 ntd ntd 64 Jul 29 10:54 6 -> /dev/pts/0

Raising the system-wide limit or the hard per-process limit

If for some reason you do need to raise the hard limit and/or the system limit, then let me know. There are other solutions for that, and they can be done without manual intervention as well.

[Delicious-Bacon]

This is a big relief. THANK YOU! ❤️

We need this documented somewhere. 🥺

[fadeevab]

What's the true way to raise the hard limit? I faced with the issue, there is a long thread full of workarounds and controversies #1688. Seems like the common case is databases which need a lot of open files.

There is the following workaround (running a command from yourself after editing the /etc/security/limits.conf and restarting the WSL2):

sudo su username --shell /bin/bash --command "ulimit -n"

[shmyer]

I had issues with raising the limit aswell, found a solution that worked for me. See my reply here: #9213 (comment)