Unexplained Crash! Help please!

Hi HA community, I am desperate and looking for your help. I feel like i’ve tried everything.

I recently transitioned my home server to a proxmox setup, which has been great. I used to run a windows 10 box with a VM for HA, but I found it resource intensive. A few months ago when my HDD failed, I decided to take the plunge and switch to proxmox.

I was able to get everything up and running pretty quickly, and love the setup.

One problem:

After a couple of weeks, HA would crash multiple times a day, requiring me to SSH into my sever and stop and start the HA VM.

I did weeks of research, and read every thread on here, proxmox forums and reddit i could find.

steps I took:

  • Remove all extraneous integrations in HA, HACS etc.
  • Remove the motioneye integration and set up it’s own separate VM (running stable for weeks now)
  • Remove the IPP integration since I’ve heard it can cause issues
    (After removing IPP, the crashes went form daily to once every ~3 days, so an improvement!)
  • Disable AdGuard integration and remove the ha instance form my routers DNS routing. (No effect)
  • Investigate chromecast device integration (Found heartbeat errors are normal)

Of course i’m not on a PI, so there are no SD card issues.

I ran into the classic logs are deleted on reboot issue, so I installed SSH, and rain a tail on the logs.

I finally caught it crashing, but don’t see anything that points at why? Maybe you guys can help me diagnose?

Logs:

2021-05-06 12:32:50 WARNING (MainThread) [homeassistant.components.light] Updating wiz_light light took longer than the scheduled update interval 0:00:30
2021-05-06 12:32:50 ERROR (MainThread) [pywizlight.bulb] [wizlight 192.168.0.139, connid 399094f49fff98] Failed to do UDP call(s) to wiz light
2021-05-06 12:32:50 ERROR (MainThread) [pywizlight.bulb] [wizlight 192.168.0.140, connid 399094f4a0015e] Failed to do UDP call(s) to wiz light
2021-05-06 12:33:30 WARNING (MainThread) [homeassistant.helpers.entity] Update of light.entry_light is taking over 10 seconds
2021-05-06 12:33:30 WARNING (MainThread) [homeassistant.helpers.entity] Update of light.nook_light is taking over 10 seconds
2021-05-06 12:33:36 ERROR (MainThread) [homeassistant.components.proxmoxve] Unexpected error fetching proxmox_coordinator data: Couldn't authenticate user: root@pam to https://192.168.0.5:8006/api2/json/access/ticket
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 173, in _async_refresh
    self.data = await self._async_update_data()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 143, in _async_update_data
    return await self.update_method()
  File "/usr/src/homeassistant/homeassistant/components/proxmoxve/__init__.py", line 175, in async_update_data
    return await hass.async_add_executor_job(poll_api)
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/proxmoxve/__init__.py", line 142, in poll_api
    vm_status = call_api_container_vm(
  File "/usr/src/homeassistant/homeassistant/components/proxmoxve/__init__.py", line 216, in call_api_container_vm
    status = proxmox.nodes(node_name).qemu(vm_id).status.current.get()
  File "/usr/local/lib/python3.8/site-packages/proxmoxer/core.py", line 105, in get
    return self(args)._request("GET", params=params)
  File "/usr/local/lib/python3.8/site-packages/proxmoxer/core.py", line 85, in _request
    resp = self._store["session"].request(method, url, data=data or None, params=params)
  File "/usr/local/lib/python3.8/site-packages/proxmoxer/backends/https.py", line 178, in request
    return super(ProxmoxHttpSession, self).request(method, url, params, data, headers, cookies, files, auth,
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 528, in request
    prep = self.prepare_request(req)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 456, in prepare_request
    p.prepare(
  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 320, in prepare
    self.prepare_auth(auth, url)
  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 551, in prepare_auth
    r = auth(self)
  File "/usr/local/lib/python3.8/site-packages/proxmoxer/backends/https.py", line 95, in __call__
    self._getNewTokens()
  File "/usr/local/lib/python3.8/site-packages/proxmoxer/backends/https.py", line 79, in _getNewTokens
    raise AuthenticationError("Couldn't authenticate user: {0} to {1}".format(self.username, self.base_url + "/access/ticket"))
proxmoxer.backends.https.AuthenticationError: Couldn't authenticate user: root@pam to https://192.168.0.5:8006/api2/json/access/ticket
2021-05-06 12:33:50 WARNING (MainThread) [homeassistant.components.light] Updating wiz_light light took longer than the scheduled update interval 0:00:30
2021-05-06 12:34:20 WARNING (MainThread) [homeassistant.components.light] Updating wiz_light light took longer than the scheduled update interval 0:00:30
2021-05-06 12:34:20 ERROR (MainThread) [pywizlight.bulb] [wizlight 192.168.0.139, connid 3990952a44d558] Failed to do UDP call(s) to wiz light
2021-05-06 12:34:20 ERROR (MainThread) [pywizlight.bulb] [wizlight 192.168.0.140, connid 3990952a44d6fc] Failed to do UDP call(s) to wiz light
2021-05-06 12:34:39 ERROR (MainThread) [homeassistant.components.proxmoxve] Unexpected error fetching proxmox_coordinator data: Couldn't authenticate user: root@pam to https://192.168.0.5:8006/api2/json/access/ticket
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 173, in _async_refresh
    self.data = await self._async_update_data()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 143, in _async_update_data
    return await self.update_method()
  File "/usr/src/homeassistant/homeassistant/components/proxmoxve/__init__.py", line 175, in async_update_data
    return await hass.async_add_executor_job(poll_api)
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/proxmoxve/__init__.py", line 142, in poll_api
    vm_status = call_api_container_vm(
  File "/usr/src/homeassistant/homeassistant/components/proxmoxve/__init__.py", line 216, in call_api_container_vm
    status = proxmox.nodes(node_name).qemu(vm_id).status.current.get()
  File "/usr/local/lib/python3.8/site-packages/proxmoxer/core.py", line 105, in get
    return self(args)._request("GET", params=params)
  File "/usr/local/lib/python3.8/site-packages/proxmoxer/core.py", line 85, in _request
    resp = self._store["session"].request(method, url, data=data or None, params=params)
  File "/usr/local/lib/python3.8/site-packages/proxmoxer/backends/https.py", line 178, in request
    return super(ProxmoxHttpSession, self).request(method, url, params, data, headers, cookies, files, auth,
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 528, in request
    prep = self.prepare_request(req)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 456, in prepare_request
    p.prepare(
  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 320, in prepare
    self.prepare_auth(auth, url)
  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 551, in prepare_auth
    r = auth(self)
  File "/usr/local/lib/python3.8/site-packages/proxmoxer/backends/https.py", line 95, in __call__
    self._getNewTokens()
  File "/usr/local/lib/python3.8/site-packages/proxmoxer/backends/https.py", line 79, in _getNewTokens
    raise AuthenticationError("Couldn't authenticate user: {0} to {1}".format(self.username, self.base_url + "/access/ticket"))
proxmoxer.backends.https.AuthenticationError: Couldn't authenticate user: root@pam to https://192.168.0.5:8006/api2/json/access/ticket
2021-05-06 12:35:02 WARNING (MainThread) [homeassistant.helpers.entity] Update of light.entry_light is taking over 10 seconds
2021-05-06 12:35:02 WARNING (MainThread) [homeassistant.helpers.entity] Update of light.nook_light is taking over 10 seconds
2021-05-06 12:35:02 WARNING (MainThread) [homeassistant.helpers.entity] Update of weather.edmonton is taking over 10 seconds
2021-05-06 12:35:23 WARNING (Thread-16) [pychromecast.socket_client] [Living Room TV(192.168.0.133):8009] Heartbeat timeout, resetting connection
2021-05-06 12:35:23 WARNING (Thread-11) [pychromecast.socket_client] [Bonus Room Speaker(192.168.0.146):8009] Heartbeat timeout, resetting connection
2021-05-06 12:35:26 WARNING (Thread-12) [pychromecast.socket_client] [Master Bedroom TV(192.168.0.145):8009] Heartbeat timeout, resetting connection

Is it the proxmox integration crashing? Have you tried removing that? It just monitors proxmox right? I never even looked at that on my test system running proxmox

I don’t think that’s the issue, as I added that integration after the crashes started, but will disable tonight and tail again!

You guys honestly rock!

Yeah, the integration just adds a sensor to show status on VM / containers. It’s so I can monitor Plex and my NVR status. (Not that they have ever crashed!)

Well the error IS related to that component.