Not enough system resources / too many files open with 33.3 upgrade

[rogersmj]

I’m starting to experience this a lot too. Currently on 0.36.1. If I am doing a lot of HA restarts, because of config changes, eventually it just stops working and throws this error, and I have to reboot. Something is not getting closed properly when the service is shut down, I’m guessing.

[RobDYI]

I haven’t got it since I rebooted back in Nov. but I haven’t don’t much configurations since then.

[sirdir1972]

I still have the same problem, too (0.37.0). I imagine it could home something to do with the webinterface, I have tons of connections to :8123, even is hass is killed - and even in established state.

[heathpitts]

I’m still seeing the exact same thing

[Sam]

+1

Observed about 20-30 web connections.

[rogersmj]

My server just died today, stopped responding to all commands. When I went to check the log file, it was this issue again.

I’m not sure what to do to troubleshoot, but would be happy to help if someone can give me some direction.

[jschwalbe (Justin S)]

Possible solution:
I ran lsof | grep 8123 and found that there were tons and tons of these sort of lines:

node       401  ob1   18u     IPv4 0xdd2a305bd36460cf        0t0      TCP localhost:62801->localhost:8123 (ESTABLISHED)
node       401  ob1   19u     IPv4 0xdd2a305bccf36cef        0t0      TCP localhost:62802->localhost:8123 (ESTABLISHED)
node       401  ob1   20u     IPv4 0xdd2a305bccf37edf        0t0      TCP localhost:62803->localhost:8123 (ESTABLISHED)
node       401  ob1   21u     IPv4 0xdd2a305bccf35aff        0t0      TCP localhost:62804->localhost:8123 (ESTABLISHED)

The only thing that was running node was homebridge. I killed and restarted homebridge and tada homeassistant was working again! Not sure why it was happening in the first place though. But it seems to be related at least in some regard to quitting and restarting hass with the connections not having a chance to close.

edited to reflect homeBRIDGE not homeBREW… oops!

[rabittn (Ryan)]

I have found this related to homebridge.

If i stop homebridge. Restart HA and then start homebridge everything plays nice.

In my setup it looks like homebridge goes on a frenzy and opens 100’s of connections to the HA API.

[Sam]

My original setup was on a Mac Mini. Gave up on that a couple of weeks ago and built loaded Hassbian on a Pi 3. No errors and install was easier. Backup of the SD card is easy so if something goes wrong I can quickly fall back to a working system.

[trymbill (Magnus)]

Experienced the same thing here last night running OS X Sierra and HASS 0.39.2. Not running homebridge and no zombie node processes running.

Hope this doesn’t become a re-occurring thing.

[moskovskiy82]

I’m egetting the same. Homeassistant 0.40 no home bridge.
Any ideas on how to increase the number of files? Raspbian?

Couldn’t get to work with MySQL reliably. Switched to Postgres. Now getting the following after 3-4 days. Anybody know how to fix this one? Guess the same reason my nmap scan is also failing

requests.exceptions.ConnectionError: HTTPConnectionPool(host='xxx', port=8086): Max retries exceeded with url: /write?db=home_assistant (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x3c5f66b0>: Failed to establish a new connection: [Errno 24] Too many open files',))

Rnin lsof see a lot of entries like this

hass       720 1061 homeassistant   55u     0000       0,11        0       4714 anon_inode
hass       720 1061 homeassistant   56u     0000       0,11        0       4714 anon_inode
hass       720 1061 homeassistant   57u     unix 0xb68eaa00      0t0      28682 socket
hass       720 1061 homeassistant   58u     unix 0xb68ea100      0t0      28683 socket
hass       720 1061 homeassistant   59u     0000       0,11        0       4714 anon_inode
hass       720 1061 homeassistant   60u     unix 0xb6964300      0t0      23403 socket
hass       720 1061 homeassistant   61u     unix 0xb9358300      0t0      33076 socket
hass       720 1061 homeassistant   62u     unix 0xb935a700      0t0      33077 socket
hass       720 1061 homeassistant   63u     0000       0,11        0       4714 anon_inode
hass       720 1061 homeassistant   64u     unix 0xb6965200      0t0      32212 socket
hass       720 1061 homeassistant   65u     unix 0xb6964600      0t0      32213 socket
hass       720 1061 homeassistant   66u     0000       0,11        0       4714 anon_inode
hass       720 1061 homeassistant   67u     0000       0,11        0       4714 anon_inode
hass       720 1061 homeassistant   68u     0000       0,11        0       4714 anon_inode
hass       720 1061 homeassistant   69u     unix 0xb935a100      0t0      48704 socket
hass       720 1061 homeassistant   70u     unix 0xb935b000      0t0      48705 socket

[maddox (Jon Maddox)]

This used to happen to me after restarting HA too many times after a reboot. I would have to reboot to fix it.

Now, since 0.40, it happens after HA is running for a while. It’s completely broken my setup. It appears macOS just has an insanely low limit for this. We just need to increase the allowed number of files to be open.

You can see the list here:

$ ulimit -a
-t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) 8192
-c: core file size (blocks) 0
-v: address space (kbytes) unlimited
-l: locked-in-memory size (kbytes) unlimited
-u: processes 709
-n: file descriptors 256

after running:

ulimit -Sn 10000

it shows

-t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) 8192
-c: core file size (blocks) 0
-v: address space (kbytes) unlimited
-l: locked-in-memory size (kbytes) unlimited
-u: processes 709
-n: file descriptors 10000

But it doesn’t stick on a reboot. It goes back to 256.

Does anyone know the best practice for getting this to stick?

[diplix (felix schwenzel)]

@maddox

Does anyone know the best practice for getting this to stick?

How to persistently control maximum system resource consumption on Mac?

https://blog.dekstroza.io/ulimit-shenanigans-on-osx-el-capitan/

[maddox (Jon Maddox)]

I found this particular solution to be the cleanest and least invasive on your system.

[mihalski (Michal Dobrzynski)]

Has anyone found a solution to this?

I’m finding HA has become totally unusable in the last few weeks due to this problem.

[gpbenton (Graham)]

This post found that Web OS TV was leaking files, so if you have that it might be worth turning off

Components that can break whole setup

Had problems configuring HA for some time. All of a sudden either postgres or mysql (tried both) would stop logging and break up with no warnings in the log. Sometimes an error like connection timed out was reported. My HA couldn’t run for more than 3 days in a row without restarting. Freezes or breaking out of other components (nmap, ping and others were failing with one error or another). Than all of a sudden i have seen a post on github HA issues about Web OS Tv opening too many connections…

[Eric_Bauer]

Had it, commented it out, still having issues.

[gpbenton (Graham)]

How about these two

Too many open files with Unifi device tracker and new LIFX lights components

I’ve been fighting a major recent problem I’ve been having with HA that results in “too many open files” errors on a MASSIVE scale… Still working to figure it out, but this seems relevant… pi@homepi:~ $ sudo lsof | wc -l 21795 pi@homepi:~ $ sudo lsof |grep homeassistant |wc -l 15445 pi@homepi:~ $ sudo lsof |grep homeassistant |grep -i lifx |wc -l 4998 pi@homepi:~ $ sudo lsof |grep homeassistant |grep -i ubnt |wc -l 8942 pi@homepi:~ $ sudo lsof |grep homeassistant |grep -vi ubnt | grep -vi lifx …

[Martinvdm (Martin)]

Exactly the same behavior today. Restart HA and Homebridge and everything fine. Perhaps it is something with the latest homebridge version?