HA keeps Crashing and running out of Memory usually around 10:30pm

comet424 · April 13, 2022, 3:10am

so i finding last few nights HA crashes itself… the UI looses connection saying reconnecting…
the Unraid console shows Out of Memory… the Supervisor logs shows its updating itself but it cant… how do i fix this? and then like 10 min goes by it fixes itself or so
first pic is the bunch of out of memory last night… tonights is the single out of memory but happens every night last few nights

22-04-12 22:40:37 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API connection is closed
22-04-12 22:40:49 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API error: Received message 257:None is not str
22-04-12 22:40:53 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API connection is closed
22-04-12 22:41:09 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API connection is closed
22-04-12 22:41:47 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state CoreState.RUNNING
22-04-12 22:41:47 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.PWNED/ContextType.ADDON
22-04-12 22:41:47 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.TRUST/ContextType.PLUGIN
22-04-12 22:42:08 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/home-assistant/addons repository
22-04-12 22:42:40 INFO (MainThread) [supervisor.api.middleware.security] /supervisor/info access from cebe7a76_hassio_google_drive_backup
22-04-12 22:42:40 INFO (MainThread) [supervisor.api.middleware.security] /backups access from cebe7a76_hassio_google_drive_backup
22-04-12 22:42:40 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json
22-04-12 22:42:40 WARNING (MainThread) [supervisor.store.git] There is already a task in progress
22-04-12 22:42:41 INFO (SyncWorker_6) [supervisor.docker.interface] Stopping hassio_observer application
22-04-12 22:42:41 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
22-04-12 22:42:41 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request initialize
22-04-12 22:42:41 INFO (MainThread) [supervisor.api.proxy] WebSocket access from c4d25c74_edge
22-04-12 22:42:41 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request running
22-04-12 22:42:41 INFO (MainThread) [supervisor.store] Loading add-ons from store: 23 all - 0 new - 0 remove
22-04-12 22:42:41 INFO (MainThread) [supervisor.store] Loading add-ons from store: 23 all - 0 new - 0 remove
22-04-12 22:42:41 ERROR (MainThread) [supervisor.utils] Can't execute check_trust while a task is in progress
22-04-12 22:42:41 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.FREE_SPACE/ContextType.SYSTEM
22-04-12 22:42:41 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.TRUST/ContextType.SUPERVISOR
22-04-12 22:42:41 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.SECURITY/ContextType.CORE
22-04-12 22:42:41 INFO (MainThread) [supervisor.resolution.checks.base] Run check for IssueType.TRUST/ContextType.CORE
22-04-12 22:42:41 INFO (MainThread) [supervisor.resolution.check] System checks complete
22-04-12 22:42:41 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.RUNNING
22-04-12 22:42:45 INFO (SyncWorker_6) [supervisor.docker.interface] Cleaning hassio_observer application
22-04-12 22:42:45 INFO (MainThread) [supervisor.plugins.observer] Starting observer plugin
22-04-12 22:42:46 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
22-04-12 22:42:46 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state CoreState.RUNNING
22-04-12 22:42:46 INFO (MainThread) [supervisor.resolution.fixups.store_execute_reset] Reset corrupt Store: core
22-04-12 22:42:47 INFO (SyncWorker_6) [supervisor.docker.observer] Starting Observer ghcr.io/home-assistant/amd64-hassio-observer with version 2021.10.0 - 172.30.32.6
22-04-12 22:42:47 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request initialize
22-04-12 22:42:47 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request initialize
22-04-12 22:42:47 INFO (MainThread) [supervisor.api.proxy] WebSocket access from core_almond
22-04-12 22:42:47 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request running
22-04-12 22:42:47 INFO (MainThread) [supervisor.api.proxy] WebSocket access from a0d7b954_nodered
22-04-12 22:42:47 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request running
22-04-12 22:42:49 ERROR (MainThread) [supervisor.store.git] Can't update https://github.com/home-assistant/addons repo: Cmd('git') failed due to: exit code(128)
  cmdline: git fetch -v --update-shallow --depth=1 origin
  stderr: 'fatal: unable to get current working directory: No such file or directory'.
22-04-12 22:42:49 INFO (MainThread) [supervisor.store.git] Cloning add-on https://github.com/home-assistant/addons repository
22-04-12 22:42:49 ERROR (MainThread) [asyncio] Task exception was never retrieved
future: <Task finished name='Task-66064' coro=<Repository.update() done, defined at /usr/src/supervisor/supervisor/store/repository.py:106> exception=StoreGitError()>
Traceback (most recent call last):
  File "/usr/src/supervisor/supervisor/store/git.py", line 155, in pull
    await self.sys_run_in_executor(
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/site-packages/git/remote.py", line 886, in fetch
    res = self._get_fetch_info_from_stderr(proc, progress,
  File "/usr/local/lib/python3.9/site-packages/git/remote.py", line 750, in _get_fetch_info_from_stderr
    proc.wait(stderr=stderr_text)
  File "/usr/local/lib/python3.9/site-packages/git/cmd.py", line 502, in wait
    raise GitCommandError(remove_password_if_present(self.args), status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git fetch -v --update-shallow --depth=1 origin
  stderr: 'fatal: unable to get current working directory: No such file or directory'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/src/supervisor/supervisor/store/repository.py", line 110, in update
    await self.git.pull()
  File "/usr/src/supervisor/supervisor/jobs/decorator.py", line 108, in wrapper
    raise err
  File "/usr/src/supervisor/supervisor/jobs/decorator.py", line 106, in wrapper
    return await self._method(*args, **kwargs)
  File "/usr/src/supervisor/supervisor/store/git.py", line 195, in pull
    raise StoreGitError() from err
supervisor.exceptions.StoreGitError
22-04-12 22:42:49 INFO (MainThread) [supervisor.store] Loading add-ons from store: 0 all - 0 new - 23 remove
22-04-12 22:42:49 INFO (MainThread) [supervisor.store] Loading add-ons from store: 0 all - 0 new - 0 remove
22-04-12 22:42:53 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete
22-04-12 22:47:48 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json
22-04-12 22:47:48 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/home-assistant/addons repository
22-04-12 22:47:48 INFO (MainThread) [supervisor.store] Loading add-ons from store: 23 all - 23 new - 0 remove
22-04-12 22:47:48 INFO (MainThread) [supervisor.store] Loading add-ons from store: 23 all - 0 new - 0 remove
22-04-12 22:52:52 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json
22-04-12 22:52:52 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/home-assistant/addons repository
22-04-12 22:52:52 INFO (MainThread) [supervisor.store] Loading add-ons from store: 23 all - 0 new - 0 remove
22-04-12 22:52:52 INFO (MainThread) [supervisor.store] Loading add-ons from store: 23 all - 0 new - 0 remove
22-04-12 22:57:57 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json
22-04-12 22:57:57 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/home-assistant/addons repository
22-04-12 22:57:57 INFO (MainThread) [supervisor.store] Loading add-ons from store: 23 all - 0 new - 0 remove
22-04-12 22:57:57 INFO (MainThread) [supervisor.store] Loading add-ons from store: 23 all - 0 new - 0 remove
22-04-12 23:03:01 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json
22-04-12 23:03:01 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/home-assistant/addons repository
22-04-12 23:03:02 INFO (MainThread) [supervisor.store] Loading add-ons from store: 23 all - 0 new - 0 remove
22-04-12 23:03:02 INFO (MainThread) [supervisor.store] Loading add-ons from store: 23 all - 0 new - 0 remove
22-04-12 23:08:06 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json
22-04-12 23:08:06 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/home-assistant/addons repository
22-04-12 23:08:06 INFO (MainThread) [supervisor.store] Loading add-ons from store: 23 all - 0 new - 0 remove
22-04-12 23:08:06 INFO (MainThread) [supervisor.store] Loading add-ons from store: 23 all - 0 new - 0 remove

KingRichard · April 13, 2022, 4:30am

Its seems your HA need more memory from your VM.

comet424 · April 13, 2022, 11:29am

didnt think i needed more then 2 gb as its not using memory guess i could give it 8
ill try 4gb first to see if it will crash tonight again

gb

do not know how to read the memory use here what its basiclly saying. as there is some blank areas and its not straight on the bottom etc

comet424 · April 16, 2022, 10:34pm

so 4 gb of ram also ran out of memory doing update… guess ill try 8 gb next… it doesnt use even 2gb seems to be a problem when it updates or so it runs out of memory

CentralCommand · April 16, 2022, 10:42pm

When you do an update, are you asking it to take a backup first? If so, two questions:

do you use the native SQLite option for db (i.e. the .db file in your /config folder)?
if #1 is yes then what’s the size of that file?

comet424 · April 16, 2022, 10:51pm

so when i mean update i mean 2 different kinds

1… When HA goes and checks for an update usually like 10:30pm at night dunno why then but thats when the HA freezes up pops up “reconnecting” on the web browser… and you get that funny memory usaage above… and the bunch of out of memory… and you get those errors in the supervisor log

2… if i click the update i do the take a backup … i do not now anymore what kinda DB im using? or the size…

3… also i have done side by side 2 windows one doing the update other in the system and it was only using 12% ram but in console says memory ran out

i looked in the config folder i dont see a db file

the one thing i know is big is that token list i cant mass delete it keeps building got more then 1000 tokens takes to o long to delete 1 at a time

patzobil · October 2, 2023, 8:11pm

I’ve got exactly the same issue everytime i upgrade the HASS OS… Each and every time the same.!

I am running an SSD Boot RPi3B, which indeed is low on RAM, however those issues don’t just occur, I had no issues with a clean install every time - till I upgrade the OS(not the Core).

Essentially the OS boots, i can get access to the dashboard etc, then in a few minutes it’s completely crashed, the dashboard doesn’t load(both local and public), the device most times remains pingable.

Only thing i havent tried yet is access to SSH(so i can get my backups over SFTP) - this is next on my list.

Did anyone have any breakthroughs on this?